Large language models (LLMs), such as ChatGPT and Copilot, are transforming software development by automating code generation and, arguably, enable rapid prototyping, support education, and boost productivity. Therefore, correctness and quality of the generated code should be on par with manually written code. To assess the current state of LLMs in generating correct code of high quality, we conducted controlled experiments with ChatGPT and Copilot: we let the LLMs generate simple algorithms in Java and Python along with the corresponding unit tests and assessed the correctness and the quality (coverage) of the generated (test) codes. We observed significant differences between the LLMs, between the languages, between algorithm and test codes, and over time. The present paper reports these results together with the experimental methods allowing repeated and comparable assessments for more algorithms, languages, and LLMs over time.

本研究解决了大语言模型在代码生成中的正确性和质量问题。通过对ChatGPT和Copilot生成Java和Python算法及其单元测试的控制实验，提出了一种可重复和可比的评估方法，最终发现了不同模型、语言和时间间隔下的显著差异。这些结果有助于进一步优化代码生成技术并提升软件开发效率。

大语言模型生成代码的研究