Large Language Models (LLMs) have demonstrated potential in cybersecurity
applications but have also caused lower confidence due to problems like
hallucinations and a lack of truthfulness. Existing benchmarks provide general
evaluations but do not sufficiently address the practical and applied aspects
of LLM performance in cybersecurity-specific tasks. To address this gap, we
introduce the SECURE (Security Extraction, Understanding \& Reasoning
Evaluation), a benchmark designed to assess LLMs performance in realistic
cybersecurity scenarios. SECURE includes six datasets focussed on the
Industrial Control System sector to evaluate knowledge extraction,
understanding, and reasoning based on industry-standard sources. Our study
evaluates seven state-of-the-art models on these tasks, providing insights into
their strengths and weaknesses in cybersecurity contexts, and offer
recommendations for improving LLMs reliability as cyber advisory tools.

大型语言模型（LLMs）在网络安全应用中显示出潜力，但也因幻觉和缺乏真实性等问题导致信心下降。现有的基准测试提供了一般评估，但不能充分解决 LLMs 在网络安全特定任务中的实际应用方面的问题。为了填补这个空白，我们引入了 SECURE（安全提取、理解与推理评估）基准测试，该基准测试旨在评估 LLMs 在现实网络安全场景中的性能。SECURE 包括六个数据集，重点关注工业控制系统领域，以评估基于行业标准来源的知识提取、理解和推理。我们的研究评估了七种最先进的模型在这些任务上的表现，为改进 LLMs 作为网络咨询工具的可靠性提供了见解和建议。