Large Language Models (LLMs) are increasingly used by software engineers for code generation. However, limitations of LLMs such as irrelevant or incorrect code have highlighted the need for prompt programming (or prompt engineering) where engineers apply specific prompt techniques (e.g., chain-of-thought or input-output examples) to improve the generated code. Despite this, the impact of different prompt techniques -- and their combinations -- on code generation remains underexplored. In this study, we introduce CodePromptEval, a dataset of 7072 prompts designed to evaluate five prompt techniques (few-shot, persona, chain-of-thought, function signature, list of packages) and their effect on the correctness, similarity, and quality of complete functions generated by three LLMs (GPT-4o, Llama3, and Mistral). Our findings show that while certain prompt techniques significantly influence the generated code, combining multiple techniques does not necessarily improve the outcome. Additionally, we observed a trade-off between correctness and quality when using prompt techniques. Our dataset and replication package enable future research on improving LLM-generated code and evaluating new prompt techniques.

本研究解决了大型语言模型在代码生成中存在的不足之处，特别是提示技术对生成代码精确性和质量的影响。我们引入了CodePromptEval数据集，评估不同提示技术对代码生成结果的影响，发现某些提示技术显著改善了生成代码的质量，但多种技术的结合并不一定提升效果，同时在正确性和质量之间存在权衡。这一研究为未来的提示技术改进和大型语言模型生成代码的研究提供了基础。

提示编程对函数级代码生成的影响