The alignment of language models with human preferences is vital for their
application in real-world tasks. The problem is formulated as optimizing the
model's policy to maximize the expected reward that reflects human preferences
with minimal deviation from the initial policy. While considered as a
straightforward solution, reinforcement learning (RL) suffers from high
variance in policy updates, which impedes efficient policy improvement.
Recently, direct preference optimization (DPO) was proposed to directly
optimize the policy from preference data. Though simple to implement, DPO is
derived based on the optimal policy that is not assured to be achieved in
practice, which undermines its convergence to the intended solution.
In this paper, we propose efficient exact optimization (EXO) of the alignment
objective. We prove that EXO is guaranteed to optimize in the same direction as
the RL algorithms asymptotically for arbitary parametrization of the policy,
while enables efficient optimization by circumventing the complexities
associated with RL algorithms. We compare our method to DPO with both
theoretical and empirical analyses, and further demonstrate the advantages of
our method over existing approaches on realistic human preference data.

我们提出了一种高效的精确优化方法 (EXO)，证明了它在与 RL 算法同向渐进地优化策略参数函数上是可保证的，并通过绕过与 RL 算法相关的复杂性来实现高效优化。我们通过理论和实证分析将我们的方法与 DPO 进行比较，并进一步展示了在现实人类偏好数据上我们方法的优势。