We propose the Data Contamination Quiz, a simple and effective approach to
detect data contamination in large language models (LLMs) and estimate the
amount of it. Specifically, we frame data contamination detection as a series
of multiple-choice questions. We devise a quiz format wherein three perturbed
versions of each dataset instance are created. These changes only include
word-level perturbations, replacing words with their contextual synonyms,
ensuring both the semantic and sentence structure remain exactly the same as
the original instance. Together with the original instance, these perturbed
versions constitute the choices in the quiz. Given that the only distinguishing
signal among these choices is the exact wording, an LLM, when tasked with
identifying the original instance from the choices, opts for the original if it
has memorized it in its pre-training phase--a trait intrinsic to LLMs. A
dataset partition is then marked as contaminated if the LLM's performance on
the quiz surpasses what random chance suggests. Our evaluation spans seven
datasets and their respective splits (train and test/validation) on two
state-of-the-art LLMs: GPT-4 and GPT-3.5. While lacking access to the
pre-training data, our results suggest that our approach not only enhances the
detection of data contamination but also provides an accurate estimation of its
extent, even when the contamination signal is weak.

我们提出了数据污染测验，一种用于检测大型语言模型（LLMs）中数据污染并估计其量的简单有效方法。我们将数据污染检测构建为一系列多项选择问题，创建了三个扰动版本的每个数据集实例。通过仅进行词级扰动，用其上下文同义词替换单词，确保其语义和句子结构与原始实例完全相同，我们设计了一个测验格式。这些扰动版本与原始实例一起构成测验中的选择项。当只有确切的措辞可以区分这些选择项时，LLM 在识别选择项中的原始实例时，如果在预训练阶段记忆了该实例，则选择原始实例，这是 LLM 固有的特性。如果 LLM 在测验中的表现超过了随机机会所示，那么数据集划分就被标记为被污染。我们的评估涵盖了七个数据集及其相应的划分（训练和测试 / 验证），在两种最先进的 LLMs：GPT-4 和 GPT-3.5 上进行。尽管无法获得预训练数据，我们的结果表明，我们的方法不仅增强了对数据污染的检测，还可以在污染信号较弱时提供准确的估计。