Amid the expanding use of pre-training data, the phenomenon of benchmark
dataset leakage has become increasingly prominent, exacerbated by opaque
training processes and the often undisclosed inclusion of supervised data in
contemporary Large Language Models (LLMs). This issue skews benchmark
effectiveness and fosters potentially unfair comparisons, impeding the field's
healthy development. To address this, we introduce a detection pipeline
utilizing Perplexity and N-gram accuracy, two simple and scalable metrics that
gauge a model's prediction precision on benchmark, to identify potential data
leakages. By analyzing 31 LLMs under the context of mathematical reasoning, we
reveal substantial instances of training even test set misuse, resulting in
potentially unfair comparisons. These findings prompt us to offer several
recommendations regarding model documentation, benchmark setup, and future
evaluations. Notably, we propose the "Benchmark Transparency Card" to encourage
clear documentation of benchmark utilization, promoting transparency and
healthy developments of LLMs. we have made our leaderboard, pipeline
implementation, and model predictions publicly available, fostering future
research.

利用简单且可扩展的 Perplexity 和 N-gram 精度两个度量指标来检测潜在数据泄漏，揭示了大规模语言模型在数学推理领域存在的数据误用问题，并提出了关于模型文档、基准设置和未来评估的几点建议，其中包括提出 “基准透明卡” 以促进透明度和语言模型的健康发展。