Reinforcement learning (RL) training is inherently unstable due to factors
such as moving targets and high gradient variance. Reinforcement Learning from
Human Feedback (RLHF) and Reinforcement Learning from AI Feedback (RLAIF) can
introduce additional difficulty. Differing preferences can complicate the
alignment process, and prediction errors in a trained reward model can become
more severe as the LLM generates unseen outputs. To enhance training
robustness, RL has adopted techniques from supervised learning, such as
ensembles and layer normalization. In this work, we improve the stability of RL
training by adapting the reverse cross entropy (RCE) from supervised learning
for noisy data to define a symmetric RL loss. We demonstrate performance
improvements across various tasks and scales. We conduct experiments in
discrete action tasks (Atari games) and continuous action space tasks (MuJoCo
benchmark and Box2D) using Symmetric A2C (SA2C) and Symmetric PPO (SPPO), with
and without added noise with especially notable performance in SPPO across
different hyperparameters. Furthermore, we validate the benefits of the
symmetric RL loss when using SPPO for large language models through improved
performance in RLHF tasks, such as IMDB positive sentiment sentiment and TL;DR
summarization tasks.

通过从有噪声数据中借鉴逆交叉熵（RCE）的方法将其适应于强化学习，定义了一种对称的 RL 损失，从而提高了 RL 训练的稳定性。在离散动作任务（Atari 游戏）和连续动作空间任务（MuJoCo 基准和 Box2D）中进行了实验，并通过改进 RLHF 任务的性能来验证对于大型语言模型使用 SPPO 的对称 RL 损失的好处，如 IMDB 正面情感和 TL;DR 摘要任务。