A common approach for aligning language models to human preferences is to
first learn a reward model from preference data, and then use this reward model
to update the language model. We study two closely related problems that arise
in this approach. First, any monotone transformation of the reward model
preserves preference ranking; is there a choice that is ``better'' than others?
Second, we often wish to align language models to multiple properties: how
should we combine multiple reward models? Using a probabilistic interpretation
of the alignment procedure, we identify a natural choice for transformation for
(the common case of) rewards learned from Bradley-Terry preference models. This
derived transformation has two important properties. First, it emphasizes
improving poorly-performing outputs, rather than outputs that already score
well. This mitigates both underfitting (where some prompts are not improved)
and reward hacking (where the model learns to exploit misspecification of the
reward model). Second, it enables principled aggregation of rewards by linking
summation to logical conjunction: the sum of transformed rewards corresponds to
the probability that the output is ``good'' in all measured properties, in a
sense we make precise. Experiments aligning language models to be both helpful
and harmless using RLHF show substantial improvements over the baseline
(non-transformed) approach.

学习从偏好数据中获取奖励模型并将其用于更新语言模型是一种常见的调整语言模型与人的偏好一致性的方法。我们研究了在这种方法中出现的两个相关问题。首先，对于奖励模型的单调转换，是否有一种选择比其他选择更好？其次，我们经常希望将语言模型与多个属性对齐：我们应该如何组合多个奖励模型？通过使用与 Bradley-Terry 偏好模型学习的奖励（常见情况）的概率解释对齐过程，我们确定了一种自然的转换选择。这个派生的转换具有两个重要属性。首先，它强调改进表现较差的输出，而不是已经得分很高的输出。这减轻了欠拟合（其中某些提示没有改进）和奖励欺骗（模型学习利用奖励模型的误规范）。其次，它通过将求和与逻辑合取链接，实现了奖励的有原则的聚合：转换后的奖励的总和对应于在所有测量属性中输出是 “好” 的概率，我们可以准确刻画这种概率。使用 RLHF 对调整语言模型以既有帮助又无害的方式进行实验，与基准（未转换）方法相比有显著改进。