Security code review aims to combine automated tools and manual efforts to
detect security defects during development. The rapid development of Large
Language Models (LLMs) has shown promising potential in software development,
as well as opening up new possibilities in automated security code review. To
explore the challenges of applying LLMs in practical code review for security
defect detection, this study compared the detection performance of three
state-of-the-art LLMs (Gemini Pro, GPT-4, and GPT-3.5) under five prompts on
549 code files that contain security defects from real-world code reviews.
Through analyzing 82 responses generated by the best-performing LLM-prompt
combination based on 100 randomly selected code files, we extracted and
categorized quality problems present in these responses into 5 themes and 16
categories. Our results indicate that the responses produced by LLMs often
suffer from verbosity, vagueness, and incompleteness, highlighting the
necessity to enhance their conciseness, understandability, and compliance to
security defect detection. This work reveals the deficiencies of LLM-generated
responses in security code review and paves the way for future optimization of
LLMs towards this task.

应用大型语言模型进行安全代码审查时，其生成的回复通常存在冗长、含糊和不完整等问题，需要提高其简练性、可理解性和安全缺陷检测的合规性。本研究比较了三种先进的大型语言模型在 549 个包含安全缺陷的真实代码审查文件上的五个提示下的检测性能，通过分析最佳性能的大型语言模型 - 提示组合产生的 82 个回复中 100 个随机选择的代码文件，提取和分类了这些回复中存在的质量问题，总结出 5 个主题和 16 个类别。该研究揭示了大型语言模型生成的回复在安全代码审查中的不足之处，并为未来优化大型语言模型以更好地完成这一任务铺平了道路。

LLMs 对安全代码审查的深入探讨

Security Code Review by LLMs: A Deep Dive into Responses

In recent years, artificial intelligence has had a conspicuous growth in
almost every aspect of life. One of the most applicable areas is security code
review, in which a lot of AI-based tools and approaches have been proposed.
Recently, ChatGPT has caught a huge amount of attention with its remarkable
performance in following instructions and providing a detailed response.
Regarding the similarities between natural language and code, in this paper, we
study the feasibility of using ChatGPT for vulnerability detection in Python
source code. Toward this goal, we feed an appropriate prompt along with
vulnerable data to ChatGPT and compare its results on two datasets with the
results of three widely used Static Application Security Testing tools (Bandit,
Semgrep and SonarQube). We implement different kinds of experiments with
ChatGPT and the results indicate that ChatGPT reduces the false positive and
false negative rates and has the potential to be used for Python source code
vulnerability detection.

使用 ChatGPT 进行 Python 源代码的漏洞检测的可行性研究，结果表明 ChatGPT 可以降低误报和漏报率，并具有潜力用于 Python 源代码的漏洞检测。