The training process of large language models (LLMs) often involves varying
degrees of test data contamination. Although current LLMs are achieving
increasingly better performance on various benchmarks, their performance in
practical applications does not always match their benchmark results. Leakage
of benchmarks can prevent the accurate assessment of LLMs' true performance.
However, constructing new benchmarks is costly, labor-intensive and still
carries the risk of leakage. Therefore, in this paper, we ask the question, Can
we reuse these leaked benchmarks for LLM evaluation? We propose Inference-Time
Decontamination (ITD) to address this issue by detecting and rewriting leaked
samples without altering their difficulties. ITD can mitigate performance
inflation caused by memorizing leaked benchmarks. Our proof-of-concept
experiments demonstrate that ITD reduces inflated accuracy by 22.9% on GSM8K
and 19.0% on MMLU. On MMLU, using Inference-time Decontamination can lead to a
decrease in the results of Phi3 and Mistral by 6.7% and 3.6% respectively. We
hope that ITD can provide more truthful evaluation results for large language
models.

我们提出了推理时间去污染（ITD）方法，通过检测和重写泄露样本而不改变它们的难度，来降低由于记忆泄露基准测试而导致的性能夸大影响。实验结果表明，ITD 方法在 GSM8K 上可以将夸大的准确性降低 22.9％，在 MMLU 上可以降低 19.0％。在 MMLU 上，使用推理时间去污染方法可以使 Phi3 和 Mistral 的结果分别降低 6.7％和 3.6％。我们希望 ITD 方法能为大型语言模型提供更真实的评估结果。