In recent years, Large Language Models (LLMs) have witnessed a remarkable surge in prevalence, altering the landscape of natural language processing and machine learning. One key factor in improving the performance of LLMs is alignment with humans achieved with Reinforcement Learning from Human Feedback (RLHF), as for many LLMs such as GPT-4, Bard, etc. In addition, recent studies are investigating the replacement of human feedback with feedback from other LLMs named Reinforcement Learning from AI Feedback (RLAIF). We examine the biases that come along with evaluating LLMs with other LLMs and take a closer look into verbosity bias -- a bias where LLMs sometimes prefer more verbose answers even if they have similar qualities. We see that in our problem setting, GPT-4 prefers longer answers more than humans. We also propose a metric to measure this bias.

近年来，大型语言模型（LLMs）在自然语言处理和机器学习领域取得了显著的发展，其性能提升的一个关键因素是通过人类反馈实现与人类的对齐，同时研究发现其他LLMs进行强化学习也可以取代人类反馈。本文研究了使用其他LLMs来评估LLMs时出现的偏差，并着重探究了冗长性偏差——即LLMs有时倾向于提供更冗长的答案，即使质量相似。我们发现在我们的问题设置中，GPT-4更倾向于提供更长的答案，我们提出了一种度量这种偏差的指标。

大型语言模型中偏重冗余的偏好标注