In recent years, Large Language Models (LLMs) have witnessed a remarkable
surge in prevalence, altering the landscape of natural language processing and
machine learning. One key factor in improving the performance of LLMs is
alignment with humans achieved with Reinforcement Learning from Human Feedback
(RLHF), as for many LLMs such as GPT-4, Bard, etc. In addition, recent studies
are investigating the replacement of human feedback with feedback from other
LLMs named Reinforcement Learning from AI Feedback (RLAIF). We examine the
biases that come along with evaluating LLMs with other LLMs and take a closer
look into verbosity bias -- a bias where LLMs sometimes prefer more verbose
answers even if they have similar qualities. We see that in our problem
setting, GPT-4 prefers longer answers more than humans. We also propose a
metric to measure this bias.

近年来，大型语言模型（LLMs）在自然语言处理和机器学习领域取得了显著的发展，其性能提升的一个关键因素是通过人类反馈实现与人类的对齐，同时研究发现其他 LLMs 进行强化学习也可以取代人类反馈。本文研究了使用其他 LLMs 来评估 LLMs 时出现的偏差，并着重探究了冗长性偏差 —— 即 LLMs 有时倾向于提供更冗长的答案，即使质量相似。我们发现在我们的问题设置中，GPT-4 更倾向于提供更长的答案，我们提出了一种度量这种偏差的指标。