How to evaluate Large Language Models (LLMs) in code generation is an open
question. Many benchmarks have been proposed but are inconsistent with
practical software projects, e.g., unreal program distributions, insufficient
dependencies, and small-scale project contexts. Thus, the capabilities of LLMs
in practical projects are still unclear. In this paper, we propose a new
benchmark named DevEval, aligned with Developers' experiences in practical
projects. DevEval is collected through a rigorous pipeline, containing 2,690
samples from 119 practical projects and covering 10 domains. Compared to
previous benchmarks, DevEval aligns to practical projects in multiple
dimensions, e.g., real program distributions, sufficient dependencies, and
enough-scale project contexts. We assess five popular LLMs on DevEval (e.g.,
gpt-4, gpt-3.5-turbo, CodeLLaMa, and StarCoder) and reveal their actual
abilities in code generation. For instance, the highest Pass@1 of gpt-3.5-turbo
only is 42 in our experiments. We also discuss the challenges and future
directions of code generation in practical projects. We open-source DevEval and
hope it can facilitate the development of code generation in practical
projects.

通过提出一个与开发者在实践项目中的经验相一致的新基准 DevEval，我们评估了五个热门的大型语言模型在代码生成方面的实际能力，揭示了它们的实际表现，并讨论了在实践项目中代码生成的挑战和未来发展方向。