Large Language Models (LLMs) are gaining popularity among software engineers.
A crucial aspect of developing effective code-generation LLMs is to evaluate
these models using a robust benchmark. Evaluation benchmarks with quality
issues can provide a false sense of performance. In this work, we conduct the
first-of-its-kind study of the quality of prompts within benchmarks used to
compare the performance of different code generation models. To conduct this
study, we analyzed 3,566 prompts from 9 code generation benchmarks to identify
quality issues in them. We also investigated whether fixing the identified
quality issues in the benchmarks' prompts affects a model's performance. We
also studied memorization issues of the evaluation dataset, which can put into
question a benchmark's trustworthiness. We found that code generation
evaluation benchmarks mainly focused on Python and coding exercises and had
very limited contextual dependencies to challenge the model. These datasets and
the developers' prompts suffer from quality issues like spelling and
grammatical errors, unclear sentences to express developers' intent, and not
using proper documentation style. Fixing all these issues in the benchmarks can
lead to a better performance for Python code generation, but not a significant
improvement was observed for Java code generation. We also found evidence that
GPT-3.5-Turbo and CodeGen-2.5 models possibly have data contamination issues.

评估大型语言模型在代码生成方面的效果时，需要使用健全的基准测试，而不严谨的评估基准会提供虚假的性能表现。本研究分析了 9 个代码生成基准中的 3,566 个提示，以确定其中的质量问题，并研究了修复这些问题对模型性能的影响。发现评估基准主要侧重于 Python 和编码练习，且缺乏上下文依赖关系，同时还存在拼写和语法错误、表达不清晰以及不符合适当文档规范等质量问题。修复这些问题可以提高 Python 代码生成的性能，但对 Java 代码生成的改进不明显。此外，还发现 GPT-3.5-Turbo 和 CodeGen-2.5 模型可能存在数据污染问题。

代码生成中使用的提示的质量评估

Quality Assessment of Prompts Used in Code Generation

Java Code Generation consists in generating automatically Java code from a
Natural Language Text. This NLP task helps in increasing programmers'
productivity by providing them with immediate solutions to the simplest and
most repetitive tasks. Code generation is a challenging task because of the
hard syntactic rules and the necessity of a deep understanding of the semantic
aspect of the programming language. Many works tried to tackle this task using
either RNN-based, or Transformer-based models. The latter achieved remarkable
advancement in the domain and they can be divided into three groups: (1)
encoder-only models, (2) decoder-only models, and (3) encoder-decoder models.
In this paper, we provide a comprehensive review of the evolution and progress
of deep learning models in Java code generation task. We focus on the most
important methods and present their merits and limitations, as well as the
objective functions used by the community. In addition, we provide a detailed
description of datasets and evaluation metrics used in the literature. Finally,
we discuss results of different models on CONCODE dataset, then propose some
future directions.

本文主要综述了深度学习模型在 Java 代码生成任务中的进展和发展，涉及到自然语言处理、基于 RNN 和 Transformer 的模型、CONCODE 数据集等方面，并对存在的方法、数据集、评估指标、未来发展等方面进行了详细介绍。