In aligning large language models (LLMs), utilizing feedback from existing
advanced AI rather than humans is an important method to scale supervisory
signals. However, it is highly challenging for AI to understand human
intentions and societal values, and provide accurate preference feedback based
on these. Current AI feedback methods rely on powerful LLMs, carefully designed
specific principles to describe human intentions, and are easily influenced by
position bias. To address these issues, we propose a self-reference-based AI
feedback framework that enables a 13B Llama2-Chat to provide high-quality
feedback under simple and general principles such as ``best for humanity``.
Specifically, we allow the AI to first respond to the user's instructions, then
generate criticism of other answers based on its own response as a reference,
and finally determine which answer better fits human preferences according to
the criticism. Additionally, we use a self-consistency method to further reduce
the impact of position bias, and employ semantic perplexity to calculate the
preference strength differences between different answers. Experimental results
show that our method enables 13B and 70B Llama2-Chat annotators to provide
high-quality preference feedback, and the policy models trained based on these
preference data achieve significant advantages in benchmark datasets through
reinforcement learning.

通过自我引用的 AI 反馈框架，允许 13B Llama2-Chat 模型以 “最适合人类” 的原则为基准，对用户指令回应进行批判，从而提供高质量的偏好反馈，并通过自洽性方法减少位置偏差的影响、使用语义困惑度来计算不同回答的偏好强度差异，实验证明该方法使 13B 和 70B Llama2-Chat 注释器能够提供高质量的偏好反馈，并且基于这些偏好数据训练的策略模型在基准数据集上通过强化学习取得了显著的优势。