Large language models achieve high performance on many but not all downstream
tasks. The interaction between pretraining data and task data is commonly
assumed to determine this variance: a task with data that is more similar to a
model's pretraining data is assumed to be easier for that model. We test
whether distributional and example-specific similarity measures (embedding-,
token- and model-based) correlate with language model performance through a
large-scale comparison of the Pile and C4 pretraining datasets with downstream
benchmarks. Similarity correlates with performance for multilingual datasets,
but in other benchmarks, we surprisingly find that similarity metrics are not
correlated with accuracy or even each other. This suggests that the
relationship between pretraining data and downstream tasks is more complex than
often assumed.

大型语言模型的性能在许多下游任务上都很高，但并非所有任务都能实现高性能。我们通过与下游基准测试的大规模比较，测试了预训练数据与任务数据之间的相似度是否与语言模型的性能相关。我们惊讶地发现，在其他基准测试中，相似度指标与准确性甚至彼此之间都没有相关性。这表明预训练数据与下游任务之间的关系比通常认为的更加复杂。