As LLMs evolve on a daily basis, there is an urgent need for a trustworthy
evaluation method that can provide robust evaluation results in a timely
fashion. Currently, as static benchmarks are prone to contamination concerns,
users tend to trust human voting platforms, such as Chatbot Arena. However,
human annotations require extensive manual efforts. To provide an automatic,
robust, and trustworthy evaluation framework, we innovatively propose the
Auto-Arena of LLMs, which automates the entire evaluation process with LLM
agents. Firstly, an examiner LLM devises queries. Then, a pair of candidate
LLMs engage in a multi-round peer-battle around the query, during which the
LLM's true performance gaps become visible. Finally, a committee of LLM judges
collectively discuss and determine the winner, which alleviates bias and
promotes fairness. In our extensive experiment on the 17 newest LLMs,
Auto-Arena shows the highest correlation with human preferences, providing a
promising alternative to human evaluation platforms.

Auto-Arena 是一个自动、稳健且可信赖的评估框架，利用 LLM 代理自动化了整个评估过程，通过候选 LLM 之间的多轮对决以及 LLM 评委小组的讨论和决策，解决了 LLMs 评估的偏见和公平性问题，在 17 个最新的 LLMs 上得到了最高的与人类偏好的相关性，为人类评估平台提供了一个有希望的替代方案。