As Large Language Models (LLMs) have become more advanced, they have outpaced
our abilities to accurately evaluate their quality. Not only is finding data to
adequately probe particular model properties difficult, but evaluating the
correctness of a model's freeform generation alone is a challenge. To address
this, many evaluations now rely on using LLMs themselves as judges to score the
quality of outputs from other LLMs. Evaluations most commonly use a single
large model like GPT4. While this method has grown in popularity, it is costly,
has been shown to introduce intramodel bias, and in this work, we find that
very large models are often unnecessary. We propose instead to evaluate models
using a Panel of LLm evaluators (PoLL). Across three distinct judge settings
and spanning six different datasets, we find that using a PoLL composed of a
larger number of smaller models outperforms a single large judge, exhibits less
intra-model bias due to its composition of disjoint model families, and does so
while being over seven times less expensive.

使用 LLm 评估员小组 (PoLL) 替代单个大模型如 GPT4 进行评估，可以在减少评估成本的情况下，展现较少内部模型偏见，并且在三种不同的评估环境和六个不同数据集上表现更好。