While large language models have achieved remarkable performance on various
code generation benchmarks, there have been growing concerns regarding
potential contamination of these benchmarks as they may be leaked into
pretraining and finetuning data. While recent work has investigated
contamination in natural language generation and understanding tasks, there has
been less extensive research into how data contamination impacts the evaluation
of code generation, which is critical for understanding the robustness and
reliability of LLMs in programming contexts. In this work, we perform a
comprehensive study of data contamination of popular code generation
benchmarks, and precisely quantify their overlap with pretraining corpus
through both surface-level and semantic-level matching. In our experiments, we
show that there are substantial overlap between popular code generation
benchmarks and open training corpus, and models perform significantly better on
the subset of the benchmarks where similar solutions are seen during training.
We also conduct extensive analysis on the factors that affects model
memorization and generalization, such as model size, problem difficulty, and
question length. We release all resulting files from our matching pipeline for
future research.

该研究综合研究了大型语言模型在代码生成任务中的数据污染问题，分析了常见代码生成基准测试与预训练语料之间的重叠程度，并揭示了类似训练解决方案出现时模型性能显著提高的现象，同时分析了模型大小、问题难度和问题长度等因素对模型记忆和泛化的影响。