Recent advancements in Large Language Models (LLMs) have led to their widespread application in automated code generation. However, these models can still generate defective code that deviates from the specification. Previous research has mainly focused on the mistakes in LLM-generated standalone functions, overlooking real-world software development situations where the successful generation of the code requires software contexts such as external dependencies. In this paper, we considered both of these code generation situations and identified a range of \textit{non-syntactic mistakes} arising from LLMs' misunderstandings of coding question specifications. Seven categories of non-syntactic mistakes were identified through extensive manual analyses, four of which were missed by previous works. To better understand these mistakes, we proposed six reasons behind these mistakes from various perspectives. Moreover, we explored the effectiveness of LLMs in detecting mistakes and their reasons. Our evaluation demonstrated that GPT-4 with the ReAct prompting technique can achieve an F1 score of up to 0.65 when identifying reasons for LLM's mistakes, such as misleading function signatures. We believe that these findings offer valuable insights into enhancing the quality of LLM-generated code.

本研究针对大型语言模型生成的代码中存在的缺陷，填补了以往仅关注独立函数错误的研究空白，分析了在实际软件开发中上下文依赖带来的非语法错误。我们识别了七种非语法错误类型，并提出了六个导致这些错误的原因，评估结果表明，GPT-4结合ReAct提示技术在识别错误原因方面的F1分数可达0.65，这为提高大型语言模型生成代码的质量提供了宝贵的见解。

深入探讨大型语言模型代码生成错误：什么和为什么？