Large Language Models (LLMs) are gaining popularity among software engineers.
A crucial aspect of developing effective code-generation LLMs is to evaluate
these models using a robust benchmark. Evaluation benchmarks with quality
issues can provide a false sense of performance. In this work, we conduct the
first-of-its-kind study of the quality of prompts within benchmarks used to
compare the performance of different code generation models. To conduct this
study, we analyzed 3,566 prompts from 9 code generation benchmarks to identify
quality issues in them. We also investigated whether fixing the identified
quality issues in the benchmarks' prompts affects a model's performance. We
also studied memorization issues of the evaluation dataset, which can put into
question a benchmark's trustworthiness. We found that code generation
evaluation benchmarks mainly focused on Python and coding exercises and had
very limited contextual dependencies to challenge the model. These datasets and
the developers' prompts suffer from quality issues like spelling and
grammatical errors, unclear sentences to express developers' intent, and not
using proper documentation style. Fixing all these issues in the benchmarks can
lead to a better performance for Python code generation, but not a significant
improvement was observed for Java code generation. We also found evidence that
GPT-3.5-Turbo and CodeGen-2.5 models possibly have data contamination issues.

评估大型语言模型在代码生成方面的效果时，需要使用健全的基准测试，而不严谨的评估基准会提供虚假的性能表现。本研究分析了 9 个代码生成基准中的 3,566 个提示，以确定其中的质量问题，并研究了修复这些问题对模型性能的影响。发现评估基准主要侧重于 Python 和编码练习，且缺乏上下文依赖关系，同时还存在拼写和语法错误、表达不清晰以及不符合适当文档规范等质量问题。修复这些问题可以提高 Python 代码生成的性能，但对 Java 代码生成的改进不明显。此外，还发现 GPT-3.5-Turbo 和 CodeGen-2.5 模型可能存在数据污染问题。

代码生成中使用的提示的质量评估

Quality Assessment of Prompts Used in Code Generation

Recent advances in AI, machine learning, and NLP have led to the development
of a new generation of Large Language Models (LLMs) that are trained on massive
amounts of data and often have trillions of parameters. Commercial applications
(e.g., ChatGPT) have made this technology available to the general public, thus
making it possible to use LLMs to produce high-quality texts for academic and
professional purposes. Schools and universities are aware of the increasing use
of AI-generated content by students and they have been researching the impact
of this new technology and its potential misuse. Educational programs in
Computer Science (CS) and related fields are particularly affected because LLMs
are also capable of generating programming code in various programming
languages. To help understand the potential impact of publicly available LLMs
in CS education, we introduce CSEPrompts, a framework with hundreds of
programming exercise prompts and multiple-choice questions retrieved from
introductory CS and programming courses. We also provide experimental results
on CSEPrompts to evaluate the performance of several LLMs with respect to
generating Python code and answering basic computer science and programming
questions.

AI 生成的内容在大规模自然语言处理模型与 CS 教育中的影响及潜在滥用的研究成果

CSEPrompts：一项初级计算机科学提示的基准

CSEPrompts: A Benchmark of Introductory Computer Science Prompts

In recent years, data has emerged as the new gold, serving as a powerful tool
for creating intelligent systems. However, procuring high-quality data remains
challenging, especially for code. To address this, we developed TinyPy
Generator, a tool that generates random Python programs using a context-free
grammar. The generated programs are guaranteed to be correct by construction.
Our system uses custom production rules (in the Backus-Naur Form (BNF) format)
to recursively generate code. This allows us to generate code with different
levels of complexity, ranging from code containing only assignments to more
complex code containing conditionals and loops. Our proposed tool enables
effortless large-scale Python code generation, beneficial for a wide range of
applications. TinyPy Generator is particularly useful in the field of machine
learning, where it can generate substantial amounts of Python code for training
Python language models. Additionally, researchers who are studying programming
languages can utilize this tool to create datasets for their experiments, which
can help validate the robustness of code interpreters or compilers. Unlike
existing research, we have open-sourced our implementation. This allows
customization according to user needs and extends potential usage to other
languages.

近年来，数据作为新的黄金崛起，成为创建智能系统的强大工具。我们开发了 TinyPy Generator，这是一个使用上下文无关文法生成随机 Python 程序的工具。生成的程序通过构建保证正确。我们的工具可以轻松地大规模生成 Python 代码，对机器学习和编程语言研究特别有用。我们的实现是开源的，可以根据用户需求进行定制，并扩展到其他语言的潜在使用。

使用上下文无关文法自动生成 Python 程序

Automatic Generation of Python Programs Using Context-Free Grammars

Program synthesis strives to generate a computer program as a solution to a
given problem specification, expressed with input-output examples or natural
language descriptions. The prevalence of large language models advances the
state-of-the-art for program synthesis, though limited training resources and
data impede open access to such models. To democratize this, we train and
release a family of large language models up to 16.1B parameters, called
CODEGEN, on natural language and programming language data, and open source the
training library JAXFORMER. We show the utility of the trained model by
demonstrating that it is competitive with the previous state-of-the-art on
zero-shot Python code generation on HumanEval. We further investigate the
multi-step paradigm for program synthesis, where a single program is factorized
into multiple prompts specifying subproblems. To this end, we construct an open
benchmark, Multi-Turn Programming Benchmark (MTPB), consisting of 115 diverse
problem sets that are factorized into multi-turn prompts. Our analysis on MTPB
shows that the same intent provided to CODEGEN in multi-turn fashion
significantly improves program synthesis over that provided as a single turn.
We make the training library JAXFORMER and model checkpoints available as open
source contribution: this https URL.

本文介绍了一种基于大型语言模型的程序合成方法，其中所训练的模型在自然语言处理和编程语言数据上具有竞争力，以及构建的开放式基准 Multi-Turn Programming Benchmark，该基准将单个程序分解为多个提示，从而显着提高了程序综合的准确性。本文提供了 JAXFORMER 的开放源码，以便更广泛地使用。