Evaluation benchmarks are the cornerstone of measuring capabilities of large
language models (LLMs), as well as driving progress in said capabilities.
Originally designed to make claims about capabilities (or lack thereof) in
fully pretrained models, evaluation benchmarks are now also extensively used to
decide between various training choices. Despite this widespread usage, we
rarely quantify the variance in our evaluation benchmarks, which dictates
whether differences in performance are meaningful. Here, we define and measure
a range of metrics geared towards measuring variance in evaluation benchmarks,
including seed variance across initialisations, and monotonicity during
training. By studying a large number of models -- both openly available and
pretrained from scratch -- we provide empirical estimates for a variety of
variance metrics, with considerations and recommendations for practitioners. We
also evaluate the utility and tradeoffs of continuous versus discrete
performance measures and explore options for better understanding and reducing
this variance. We find that simple changes, such as framing choice tasks (like
MMLU) as completion tasks, can often reduce variance for smaller scale
($\sim$7B) models, while more involved methods inspired from human testing
literature (such as item analysis and item response theory) struggle to
meaningfully reduce variance. Overall, our work provides insights into variance
in evaluation benchmarks, suggests LM-specific techniques to reduce variance,
and more generally encourages practitioners to carefully factor in variance
when comparing models.

通过定义和衡量一系列度量评估基准中的方差，包括初始化中的随机性和训练过程中的单调性，本研究提供了对各种方差度量的实证估计，并为从业者提供了考虑和建议。研究发现，简单的改变策略任务（如 MMLU）为较小规模（约 7B）的模型降低了方差，而受人类测试文献启发的更复杂方法（如项目分析和项目反应理论）在降低方差方面效果不明显。总体而言，本研究通过对评估基准中的方差提供了洞察，提出了降低方差的语言模型特定技术，并鼓励从业者在比较模型时谨慎考虑方差。