Direct preference optimization (DPO) is a successful fine-tuning strategy for
aligning large language models with human preferences without the need to train
a reward model or employ reinforcement learning. DPO, as originally formulated,
relies on binary preference data and fine-tunes a language model to increase
the likelihood of a preferred response over a dispreferred response. However,
not all preference pairs are equal: while in some cases the preferred response
is only slightly better than the dispreferred response, there can be a stronger
preference for one response when, for example, the other response includes
harmful or toxic content. In this paper, we propose a generalization of DPO,
termed DPO with an offset (ODPO), that does not treat every preference pair
equally during fine-tuning. Intuitively, ODPO requires the difference between
the likelihood of the preferred and dispreferred response to be greater than an
offset value. The offset is determined based on the extent to which one
response is preferred over another. Our experiments on various tasks suggest
that ODPO significantly outperforms DPO in aligning language models, especially
when the number of preference pairs is limited.

直接偏好优化（DPO）是一种成功调优策略，用于将大型语言模型与人类偏好对齐，而无需训练奖励模型或使用强化学习。本文提出了一种名为带有偏移量的 DPO（ODPO）的 DPO 泛化方法，通过对更喜欢或更讨厌的回复之间的可能性差异设置偏移量，以在调优过程中有选择地处理偏好对。实验结果表明，ODPO 在对齐语言模型方面明显优于传统的 DPO 方法，尤其是在偏好对数量有限的情况下。