Reinforcement Learning with Human Feedback (RLHF) has achieved great success
in aligning large language models (LLMs) with human preferences. Prevalent RLHF
approaches are reward-based, following the Bradley-Terry (BT) model assumption,
which may not fully capture the complexity of human preferences. In this paper,
we explore RLHF under a general preference framework and approach it from a
game-theoretic perspective. Specifically, we formulate the problem as a
two-player game and propose a novel algorithm, iterative Nash policy
optimization (INPO). The key idea is to let the policy play against itself via
no-regret learning, thereby approximating the Nash policy. Unlike previous
methods, INPO bypasses the need for estimating the expected win rate for
individual responses, which typically incurs high computational or annotation
costs. Instead, we introduce a new loss objective that is directly minimized
over a preference dataset. We provide theoretical analysis for our approach and
demonstrate its effectiveness through experiments on various representative
benchmarks. With an LLaMA-3-8B-based SFT model, INPO achieves a 41.5%
length-controlled win rate on AlpacaEval 2.0 and a 38.3% win rate on
Arena-Hard, showing substantial improvement over the state-of-the-art iterative
algorithm [Dong et al., 2024] under the BT model assumption. Additionally, our
ablation study highlights the benefits of incorporating KL regularization for
response length control.

本文以游戏理论的角度，探讨了一种基于人类偏好与强化学习的算法，通过采用无懊悔学习，自我对抗的方式逼近纳什平衡策略，从而解决了大规模语言模型对人类反馈的规模性计算问题。实验证明该算法相较于之前的方法在不考虑个体响应的预期胜率的情况下能够取得更好的结果。