Both online and offline RLHF methods such as PPO and DPO have been extremely successful in aligning AI with human preferences. Despite their success, the existing methods suffer from a fundamental problem that their optimal solution is highly task-dependent (i.e., not robust to out-of-distribution (OOD) tasks). Here we address this challenge by proposing Self-Improving Robust Preference Optimization SRPO, a practical and mathematically principled offline RLHF framework that is completely robust to the changes in the task. The key idea of SRPO is to cast the problem of learning from human preferences as a self-improvement process, which can be mathematically expressed in terms of a min-max objective that aims at joint optimization of self-improvement policy and the generative policy in an adversarial fashion. The solution for this optimization problem is independent of the training task and thus it is robust to its changes. We then show that this objective can be re-expressed in the form of a non-adversarial offline loss which can be optimized using standard supervised optimization techniques at scale without any need for reward model and online inference. We show the effectiveness of SRPO in terms of AI Win-Rate (WR) against human (GOLD) completions. In particular, when SRPO is evaluated on the OOD XSUM dataset, it outperforms the celebrated DPO by a clear margin of 15% after 5 self-revisions, achieving WR of 90%.

提出了一种完全适应任务变化的离线强化学习方法，称为自我改进的稳健偏好优化（SRPO），通过将问题表示为自我改进过程的极小极大目标，以敌对方式联合优化自我改进策略和生成策略，并通过标准的监督优化技术以大规模进行优化，而无需奖励模型和在线推断。实验证明SRPO在ODD XSUM数据集上的效果优于DPO，经过5次自我修订后，其AI获胜率（WR）达到90%，超过DPO 15%。

自我改进的健壮偏好优化