Large Language Models (LLMs) have training corpora containing large amounts
of program code, greatly improving the model's code comprehension and
generation capabilities. However, sound comprehensive research on detecting
program vulnerabilities, a more specific task related to code, and evaluating
the performance of LLMs in this more specialized scenario is still lacking. To
address common challenges in vulnerability analysis, our study introduces a new
benchmark, VulDetectBench, specifically designed to assess the vulnerability
detection capabilities of LLMs. The benchmark comprehensively evaluates LLM's
ability to identify, classify, and locate vulnerabilities through five tasks of
increasing difficulty. We evaluate the performance of 17 models (both open- and
closed-source) and find that while existing models can achieve over 80%
accuracy on tasks related to vulnerability identification and classification,
they still fall short on specific, more detailed vulnerability analysis tasks,
with less than 30% accuracy, making it difficult to provide valuable auxiliary
information for professional vulnerability mining. Our benchmark effectively
evaluates the capabilities of various LLMs at different levels in the specific
task of vulnerability detection, providing a foundation for future research and
improvements in this critical area of code security. VulDetectBench is publicly
available at this https URL

通过创建一个新的基准测试集 VulDetectBench，我们评估了各类大型语言模型在漏洞检测任务中的性能，结果显示现有模型在识别和分类与漏洞相关的任务上可以达到 80% 以上的准确率，但在更复杂的漏洞分析任务上只能达到不到 30% 的准确率，难以为专业漏洞挖掘提供有价值的辅助信息。这个基准测试集为未来代码安全领域的研究和改进提供了基础。