Evaluating large language models (LLMs) is challenging. Traditional
ground-truth-based benchmarks fail to capture the comprehensiveness and nuance
of real-world queries, while LLM-as-judge benchmarks suffer from grading biases
and limited query quantity. Both of them may also become contaminated over
time. User-facing evaluation, such as Chatbot Arena, provides reliable signals
but is costly and slow. In this work, we propose MixEval, a new paradigm for
establishing efficient, gold-standard LLM evaluation by strategically mixing
off-the-shelf benchmarks. It bridges (1) comprehensive and well-distributed
real-world user queries and (2) efficient and fairly-graded ground-truth-based
benchmarks, by matching queries mined from the web with similar queries from
existing benchmarks. Based on MixEval, we further build MixEval-Hard, which
offers more room for model improvement. Our benchmarks' advantages lie in (1) a
0.96 model ranking correlation with Chatbot Arena arising from the highly
impartial query distribution and grading mechanism, (2) fast, cheap, and
reproducible execution (6% of the time and cost of MMLU), and (3) dynamic
evaluation enabled by the rapid and stable data update pipeline. We provide
extensive meta-evaluation and analysis for our and existing LLM benchmarks to
deepen the community's understanding of LLM evaluation and guide future
research directions.

提出了一种新的评估大型语言模型的方法 MixEval，通过混合现有的基准测试以匹配来自网络的查询与现有基准测试中的相似查询，建立了有效和可靠的 LLM 评估标准，进一步构建了 MixEval-Hard，为模型改进提供了更大的空间。