In this paper, we introduce SecQA, a novel dataset tailored for evaluating
the performance of Large Language Models (LLMs) in the domain of computer
security. Utilizing multiple-choice questions generated by GPT-4 based on the
"Computer Systems Security: Planning for Success" textbook, SecQA aims to
assess LLMs' understanding and application of security principles. We detail
the structure and intent of SecQA, which includes two versions of increasing
complexity, to provide a concise evaluation across various difficulty levels.
Additionally, we present an extensive evaluation of prominent LLMs, including
GPT-3.5-Turbo, GPT-4, Llama-2, Vicuna, Mistral, and Zephyr models, using both
0-shot and 5-shot learning settings. Our results, encapsulated in the SecQA v1
and v2 datasets, highlight the varying capabilities and limitations of these
models in the computer security context. This study not only offers insights
into the current state of LLMs in understanding security-related content but
also establishes SecQA as a benchmark for future advancements in this critical
research area.

本文介绍了 SecQA 数据集，该数据集旨在评估大型语言模型（LLMs）在计算机安全领域的性能。我们详细说明了 SecQA 的结构和目的，包括两个难度逐渐增加的版本，以提供对各种难度水平的简明评估。此外，我们使用 0-shot 和 5-shot 学习设置对 GPT-3.5-Turbo，GPT-4，Llama-2，Vicuna，Mistral 和 Zephyr 模型进行了广泛评估。我们的结果突显了这些模型在计算机安全上的不同能力和局限性。这项研究不仅提供了对 LLMs 在理解安全相关内容方面的当前状态的洞察，还将 SecQA 确立为未来在这一关键研究领域中进行进一步发展的基准。