Ensuring alignment of language models' outputs with human preferences is
critical to guarantee a useful, safe, and pleasant user experience. Thus, human
alignment has been extensively studied recently and several methods such as
Reinforcement Learning from Human Feedback (RLHF), Direct Policy Optimisation
(DPO) and Sequence Likelihood Calibration (SLiC) have emerged. In this paper,
our contribution is two-fold. First, we show the equivalence between two recent
alignment methods, namely Identity Policy Optimisation (IPO) and Nash Mirror
Descent (Nash-MD). Second, we introduce a generalisation of IPO, named IPO-MD,
that leverages the regularised sampling approach proposed by Nash-MD.
This equivalence may seem surprising at first sight, since IPO is an offline
method whereas Nash-MD is an online method using a preference model. However,
this equivalence can be proven when we consider the online version of IPO, that
is when both generations are sampled by the online policy and annotated by a
trained preference model. Optimising the IPO loss with such a stream of data
becomes then equivalent to finding the Nash equilibrium of the preference model
through self-play. Building on this equivalence, we introduce the IPO-MD
algorithm that generates data with a mixture policy (between the online and
reference policy) similarly as the general Nash-MD algorithm. We compare
online-IPO and IPO-MD to different online versions of existing losses on
preference data such as DPO and SLiC on a summarisation task.

确保语言模型输出与人类偏好的一致性对于保证用户的有用、安全和愉快的体验至关重要。本文中，我们展示了两种最近对齐方法：Identity Policy Optimisation (IPO) 和 Nash Mirror Descent (Nash-MD) 之间的等价性，并引入了一种名为 IPO-MD 的 IPO 推广算法，它利用了 Nash-MD 提出的正则化抽样方法。