Language models pre-trained on web-scale corpora demonstrate impressive
capabilities on diverse downstream tasks. However, there is increasing concern
whether such capabilities might arise from evaluation datasets being included
in the pre-training corpus -- a phenomenon known as \textit{data contamination}
-- in a manner that artificially increases performance. There has been little
understanding of how this potential contamination might influence LMs'
performance on downstream tasks. In this paper, we explore the impact of data
contamination at the pre-training stage by pre-training a series of GPT-2
models \textit{from scratch}. We highlight the effect of both text
contamination (\textit{i.e.}\ input text of the evaluation samples) and
ground-truth contamination (\textit{i.e.}\ the prompts asked on the input and
the desired outputs) from evaluation data. We also investigate the effects of
repeating contamination for various downstream tasks. Additionally, we examine
the prevailing n-gram-based definitions of contamination within current LLM
reports, pinpointing their limitations and inadequacy. Our findings offer new
insights into data contamination's effects on language model capabilities and
underscore the need for independent, comprehensive contamination assessments in
LLM studies.

本文研究了语言模型在预训练阶段遭受数据污染的影响，探索了文本污染和真实数据污染对模型性能的影响，调查了不同下游任务中污染的重复效应，并指出了当前 LLM 报告中关于污染定义的局限性和不足之处。研究结果为了解数据污染对语言模型的影响提供了新的见解，并强调在 LLM 研究中进行独立、全面的污染评估的必要性。