Learning from preference feedback has emerged as an essential step for
improving the generation quality and performance of modern language models
(LMs). Despite its widespread use, the way preference-based learning is applied
varies wildly, with differing data, learning algorithms, and evaluations used,
making disentangling the impact of each aspect difficult. In this work, we
identify four core aspects of preference-based learning: preference data,
learning algorithm, reward model, and policy training prompts, systematically
investigate the impact of these components on downstream model performance, and
suggest a recipe for strong learning for preference feedback. Our findings
indicate that all aspects are important for performance, with better preference
data leading to the largest improvements, followed by the choice of learning
algorithm, the use of improved reward models, and finally the use of additional
unlabeled prompts for policy training. Notably, PPO outperforms DPO by up to
2.5% in math and 1.2% in general domains. High-quality preference data leads to
improvements of up to 8% in instruction following and truthfulness. Despite
significant gains of up to 5% in mathematical evaluation when scaling up reward
models, we surprisingly observe marginal improvements in other categories.
We publicly release the code used for training
(this https URL) and evaluating
(this https URL) our models, along with the models
and datasets themselves
(this https URL).

从偏好反馈学习的四个核心方面：偏好数据，学习算法，奖励模型和政策训练提示，我们系统地研究了这些组件对下游模型性能的影响，并提出了一种强偏好反馈学习的方法。研究发现，偏好数据、学习算法、改进的奖励模型和额外的无标签提示对模型性能都很重要。