Despite their sophisticated capabilities, large language models (LLMs)
encounter a major hurdle in effective assessment. This paper first revisits the
prevalent evaluation method-multiple choice question answering (MCQA), which
allows for straightforward accuracy measurement. Through a comprehensive
evaluation of 24 models across 11 benchmarks, we highlight several potential
drawbacks of MCQA, for instance, the inconsistency between the MCQA evaluation
and the generation of open-ended responses in practical scenarios. In response,
we introduce an RWQ-Elo rating system, engaging 24 LLMs such as GPT-4, GPT-3.5,
Google-Gemini-Pro and LLaMA-1/-2, in a two-player competitive format, with
GPT-4 serving as the judge. Each LLM receives an Elo rating thereafter. This
system is designed to mirror real-world usage, and for this purpose, we have
compiled a new benchmark called ``Real-world questions'' (RWQ), comprising
20,772 authentic user inquiries. Additionally, we thoroughly analyze the
characteristics of our system and compare it with prior leaderboards like
AlpacaEval and MT-Bench. Our analysis reveals the stability of our RWQ-Elo
system, the feasibility of registering new models, and its potential to reshape
LLM leaderboards.

通过对 24 种模型的 11 个评估标准进行综合评估，本文首先回顾了当前的评估方法 —— 多项选择题回答（MCQA），并突出了 MCQA 的一些潜在缺点，接着引入了 RWQ-Elo 评分系统，通过 24 种大型语言模型的竞争性对战，模拟现实世界的使用情景，最后分析了系统特点、与先前排行榜的对比，揭示了 RWQ-Elo 系统的稳定性、注册新模型的可行性和其重塑 LLM 排行榜的潜力。