Large Language Models (LLMs) have shown their impressive capabilities, while
also raising concerns about the data contamination problems due to privacy
issues and leakage of benchmark datasets in the pre-training phase. Therefore,
it is vital to detect the contamination by checking whether an LLM has been
pre-trained on the target texts. Recent studies focus on the generated texts
and compute perplexities, which are superficial features and not reliable. In
this study, we propose to utilize the probing technique for pre-training data
detection by examining the model's internal activations. Our method is simple
and effective and leads to more trustworthy pre-training data detection.
Additionally, we propose ArxivMIA, a new challenging benchmark comprising arxiv
abstracts from Computer Science and Mathematics categories. Our experiments
demonstrate that our method outperforms all baselines, and achieves
state-of-the-art performance on both WikiMIA and ArxivMIA, with additional
experiments confirming its efficacy (Our code and dataset are available at
this https URL).

大语言模型（LLMs）在展示其出色的能力的同时，也引发了由于隐私问题和基准数据集泄露所导致的数据污染问题。因此，在预训练阶段检测污染是否有 LLM 在目标文本上进行了预训练变得至关重要。最近的研究关注生成的文本并计算困惑度，但这些都是表面特征而不可靠。本研究提出利用探测技术来检测预训练数据，通过检查模型的内部激活状态。我们的方法简单且有效，并实现了更可靠的预训练数据检测。此外，我们还提出了 ArxivMIA 基准，它包括来自计算机科学和数学类别的 arxiv 摘要。我们的实验证明我们的方法胜过所有基准，实现了 WikiMIA 和 ArxivMIA 的最新性能，并通过额外的实验验证了其有效性。

对预训练数据检测的语言模型探究

Probing Language Models for Pre-training Data Detection

The problem of pre-training data detection for large language models (LLMs)
has received growing attention due to its implications in critical issues like
copyright violation and test data contamination. The current state-of-the-art
approach, Min-K%, measures the raw token probability which we argue may not be
the most informative signal. Instead, we propose Min-K%++ to normalize the
token probability with statistics of the categorical distribution over the
whole vocabulary, which accurately reflects the relative likelihood of the
target token compared with other candidate tokens in the vocabulary.
Theoretically, we back up our method by showing that the statistic it estimates
is explicitly optimized during LLM training, thus serving as a reliable
indicator for detecting training data. Empirically, on the WikiMIA benchmark,
Min-K%++ outperforms the SOTA Min-K% by 6.2% to 10.5% in detection AUROC
averaged over five models. On the more challenging MIMIR benchmark, Min-K%++
consistently improves upon Min-K% and performs on par with reference-based
method, despite not requiring an extra reference model.

通过使用整个词汇表的分类分布统计规范化目标词汇的概率，Min-K%++ 相对于 Min-K% 在检测预训练数据方面表现出更高的性能，且无需额外的参考模型。