Reinforcement Learning from Human Feedback (RLHF) is a key method for
aligning large language models (LLMs) with human preferences. However, current
offline alignment approaches like DPO, IPO, and SLiC rely heavily on fixed
preference datasets, which can lead to sub-optimal performance. On the other
hand, recent literature has focused on designing online RLHF methods but still
lacks a unified conceptual formulation and suffers from distribution shift
issues. To address this, we establish that online LLM alignment is underpinned
by bilevel optimization. By reducing this formulation to an efficient
single-level first-order method (using the reward-policy equivalence), our
approach generates new samples and iteratively refines model alignment by
exploring responses and regulating preference labels. In doing so, we permit
alignment methods to operate in an online and self-improving manner, as well as
generalize prior online RLHF methods as special cases. Compared to
state-of-the-art iterative RLHF methods, our approach significantly improves
alignment performance on open-sourced datasets with minimal computational
overhead.

在线学习对于语言模型的对齐与优化是至关重要的，本文提出了一种基于双层优化的在线对齐方法，并通过探索回应和调节偏好标签来迭代生成新样本和改进模型对齐效果，以实现自我提升和广义化先前的在线学习方法。这种方法相比于现有的迭代在线学习方法，在开源数据集上显著提高了对齐性能，并具有极小的计算开销。