Large Language Model (LLM) leaderboards based on benchmark rankings are regularly used to guide practitioners in model selection. Often, the published leaderboard rankings are taken at face value - we show this is a (potentially costly) mistake. Under existing leaderboards, the relative performance of LLMs is highly sensitive to (often minute) details. We show that for popular multiple choice question benchmarks (e.g. MMLU) minor perturbations to the benchmark, such as changing the order of choices or the method of answer selection, result in changes in rankings up to 8 positions. We explain this phenomenon by conducting systematic experiments over three broad categories of benchmark perturbations and identifying the sources of this behavior. Our analysis results in several best-practice recommendations, including the advantage of a hybrid scoring method for answer selection. Our study highlights the dangers of relying on simple benchmark evaluations and charts the path for more robust evaluation schemes on the existing benchmarks.

利用基准排名构建的大型语言模型（LLM）排行榜经常被用来指导实践者选择模型，但我们展示了这是一个（潜在的昂贵）错误。在现有的排行榜中，LLM的相对性能对（通常微小的）细节非常敏感。我们通过对多个选择题基准进行实验，比如改变选项的顺序或答案选择方法，我们发现这些微小扰动会导致排名变化高达8个位置。我们通过对三类基准扰动进行系统性实验并确定其来源来解释这个现象。我们的分析得出了一些最佳实践建议，包括使用混合评分方法进行答案选择。我们的研究强调了依赖简单基准评估的风险，并规划了更健壮的评估方案的路径。

当基准成为目标：揭示大型语言模型排行榜的敏感性