Direct preference optimization (DPO) is a successful fine-tuning strategy for
aligning large language models with human preferences without the need to train
a reward model or employ reinforcement learning. DPO, as originally formulated,
relies on binary preference data and fine-tunes a language model to increase
the likelihood of a preferred response over a dispreferred response. However,
not all preference pairs are equal: while in some cases the preferred response
is only slightly better than the dispreferred response, there can be a stronger
preference for one response when, for example, the other response includes
harmful or toxic content. In this paper, we propose a generalization of DPO,
termed DPO with an offset (ODPO), that does not treat every preference pair
equally during fine-tuning. Intuitively, ODPO requires the difference between
the likelihood of the preferred and dispreferred response to be greater than an
offset value. The offset is determined based on the extent to which one
response is preferred over another. Our experiments on various tasks suggest
that ODPO significantly outperforms DPO in aligning language models, especially
when the number of preference pairs is limited.

直接偏好优化（DPO）是一种成功调优策略，用于将大型语言模型与人类偏好对齐，而无需训练奖励模型或使用强化学习。本文提出了一种名为带有偏移量的 DPO（ODPO）的 DPO 泛化方法，通过对更喜欢或更讨厌的回复之间的可能性差异设置偏移量，以在调优过程中有选择地处理偏好对。实验结果表明，ODPO 在对齐语言模型方面明显优于传统的 DPO 方法，尤其是在偏好对数量有限的情况下。

使用偏移量的直接偏好优化

Direct Preference Optimization with an Offset

A common approach for aligning language models to human preferences is to
first learn a reward model from preference data, and then use this reward model
to update the language model. We study two closely related problems that arise
in this approach. First, any monotone transformation of the reward model
preserves preference ranking; is there a choice that is ``better'' than others?
Second, we often wish to align language models to multiple properties: how
should we combine multiple reward models? Using a probabilistic interpretation
of the alignment procedure, we identify a natural choice for transformation for
(the common case of) rewards learned from Bradley-Terry preference models. This
derived transformation has two important properties. First, it emphasizes
improving poorly-performing outputs, rather than outputs that already score
well. This mitigates both underfitting (where some prompts are not improved)
and reward hacking (where the model learns to exploit misspecification of the
reward model). Second, it enables principled aggregation of rewards by linking
summation to logical conjunction: the sum of transformed rewards corresponds to
the probability that the output is ``good'' in all measured properties, in a
sense we make precise. Experiments aligning language models to be both helpful
and harmless using RLHF show substantial improvements over the baseline
(non-transformed) approach.

学习从偏好数据中获取奖励模型并将其用于更新语言模型是一种常见的调整语言模型与人的偏好一致性的方法。我们研究了在这种方法中出现的两个相关问题。首先，对于奖励模型的单调转换，是否有一种选择比其他选择更好？其次，我们经常希望将语言模型与多个属性对齐：我们应该如何组合多个奖励模型？通过使用与 Bradley-Terry 偏好模型学习的奖励（常见情况）的概率解释对齐过程，我们确定了一种自然的转换选择。这个派生的转换具有两个重要属性。首先，它强调改进表现较差的输出，而不是已经得分很高的输出。这减轻了欠拟合（其中某些提示没有改进）和奖励欺骗（模型学习利用奖励模型的误规范）。其次，它通过将求和与逻辑合取链接，实现了奖励的有原则的聚合：转换后的奖励的总和对应于在所有测量属性中输出是 “好” 的概率，我们可以准确刻画这种概率。使用 RLHF 对调整语言模型以既有帮助又无害的方式进行实验，与基准（未转换）方法相比有显著改进。