The rapid advancement of Large Language Models (LLMs) necessitates robust and
challenging benchmarks. Leaderboards like Chatbot Arena rank LLMs based on how
well their responses align with human preferences. However, many tasks such as
those related to emotional intelligence, creative writing, or persuasiveness,
are highly subjective and often lack majoritarian human agreement. Judges may
have irreconcilable disagreements about what constitutes a better response. To
address the challenge of ranking LLMs on highly subjective tasks, we propose a
novel benchmarking framework, the Language Model Council (LMC). The LMC
operates through a democratic process to: 1) formulate a test set through equal
participation, 2) administer the test among council members, and 3) evaluate
responses as a collective jury. We deploy a council of 20 newest LLMs on an
open-ended emotional intelligence task: responding to interpersonal dilemmas.
Our results show that the LMC produces rankings that are more separable,
robust, and less biased than those from any individual LLM judge, and is more
consistent with a human-established leaderboard compared to other benchmarks.

通过民主过程提出了一种新的基准框架 —— 语言模型委员会（LMC），用于在高度主观的任务中对大型语言模型进行排名，通过平等参与来制定测试集，通过集体评估响应来产生更可分离、强健且较少偏差的排名。