Reinforcement learning from human feedback (RLHF) has emerged as a reliable approach to aligning large language models (LLMs) to human preferences. Among the plethora of RLHF techniques, proximal policy optimization (PPO) is of the most widely used methods. Despite its popularity, however, PPO may suffer from mode collapse, instability, and poor sample efficiency. We show that these issues can be alleviated by a novel algorithm that we refer to as Advantage-Induced Policy Alignment (APA), which leverages a squared error loss function based on the estimated advantages. We demonstrate empirically that APA consistently outperforms PPO in language tasks by a large margin, when a separate reward model is employed as the evaluator. In addition, compared with PPO, APA offers a more stable form of control over the deviation from the model's initial policy, ensuring that the model improves its performance without collapsing to deterministic output. In addition to empirical results, we also provide a theoretical justification supporting the design of our loss function.

本研究提出了一种新算法 APA，利用估计的优势建立基于平方误差损失函数的算法进行优化，证明在使用单独的奖励模型作为评估器时，APA明显优于PPO，并且在控制模型初始策略与改进性能之间提供更稳定的形式控制，避免了模式崩溃、不稳定性和样本效率低等问题。

利用优势引导的策略对齐对语言模型进行微调