Automatic evaluation methods based on large language models (LLMs) are emerging as the standard tool for assessing the instruction-following abilities of LLM-based agents. The most common method in this paradigm, pairwise comparisons with a baseline model, critically depends on the assumption of transitive preferences. However, the validity of this assumption remains largely unexplored. In this study, we investigate the presence of non-transitivity within the AlpacaEval framework and analyze its effects on model rankings. We find that LLM judges exhibit non-transitive preferences, leading to rankings that are sensitive to the choice of the baseline model. To mitigate this issue, we show that round-robin tournaments combined with Bradley-Terry models of preference can produce more reliable rankings. Notably, our method increases both the Spearman correlation and the Kendall correlation with Chatbot Arena (95.0% -> 96.4% and 82.1% -> 86.3% respectively). To address the computational cost of round-robin tournaments, we propose Swiss-Wise Iterative Matchmaking (Swim) tournaments, using a dynamic matching strategy to capture the benefits of round-robin tournaments while maintaining computational efficiency.

本研究探讨了基于大型语言模型（LLMs）的自动评估方法中的非传递性问题，特别是在AlpacaEval框架下。我们发现LLM评审存在非传递偏好，这导致模型排名对基准模型的选择敏感。为解决这一问题，我们提出结合循环赛和Bradley-Terry偏好模型的方法，从而提高排名的可靠性，并且引入瑞士式迭代配对（Swim）以兼顾效率。

研究LLM作为评审的非传递性