The widespread adoption of conversational LLMs for software development has raised new security concerns regarding the safety of LLM-generated content. Our motivational study outlines ChatGPT's potential in volunteering context-specific information to the developers, promoting safe coding practices. Motivated by this finding, we conduct a study to evaluate the degree of security awareness exhibited by three prominent LLMs: Claude 3, GPT-4, and Llama 3. We prompt these LLMs with Stack Overflow questions that contain vulnerable code to evaluate whether they merely provide answers to the questions or if they also warn users about the insecure code, thereby demonstrating a degree of security awareness. Further, we assess whether LLM responses provide information about the causes, exploits, and the potential fixes of the vulnerability, to help raise users' awareness. Our findings show that all three models struggle to accurately detect and warn users about vulnerabilities, achieving a detection rate of only 12.6% to 40% across our datasets. We also observe that the LLMs tend to identify certain types of vulnerabilities related to sensitive information exposure and improper input neutralization much more frequently than other types, such as those involving external control of file names or paths. Furthermore, when LLMs do issue security warnings, they often provide more information on the causes, exploits, and fixes of vulnerabilities compared to Stack Overflow responses. Finally, we provide an in-depth discussion on the implications of our findings and present a CLI-based prompting tool that can be used to generate significantly more secure LLM responses.

本研究探讨了当前流行的对话式大型语言模型（LLM）在软件开发中对安全性的关注，特别是在生成内容时的潜在安全风险。通过评估Claude 3、GPT-4和Llama 3对含有脆弱代码的Stack Overflow提问的反应，我们发现这些模型在识别和警告用户有关安全漏洞方面的能力有限，且在发出警告时往往能提供更多关于漏洞原因和修复的信息。这项研究的发现揭示了LLM在安全意识方面的不足，并提出了一种生成更安全回应的工具。 

大型语言模型是否考虑安全性？对编程问题响应的实证研究