Aligning with human preference datasets has been critical to the success of
large language models (LLMs). Reinforcement learning from human feedback (RLHF)
employs a costly reward model to provide feedback for on-policy sampling
responses. Recently, offline methods that directly fit responses with binary
preferences in the dataset have emerged as alternatives. However, existing
methods do not explicitly model preference strength information, which is
crucial for distinguishing different response pairs. To overcome this
limitation, we propose Online Self-Preferring (OSP) language models to learn
from self-generated response pairs and self-judged preference strengths. For
each prompt and corresponding self-generated responses, we introduce a ranked
pairing method to construct multiple response pairs with preference strength
information. We then propose the soft-preference cross-entropy loss to leverage
such information. Empirically, we demonstrate that leveraging preference
strength is crucial for avoiding overfitting and enhancing alignment
performance. OSP achieves state-of-the-art alignment performance across various
metrics in two widely used human preference datasets. OSP is
parameter-efficient and more robust than the dominant online method, RLHF when
limited offline data are available and generalizing to out-of-domain tasks.
Moreover, OSP language models established by LLMs with proficiency in
self-preferring can efficiently self-improve without external supervision.

借助自主生成的响应对和自我评定的偏好强度信息，我们提出了在线自主偏好（OSP）语言模型来学习，展示了利用偏好强度是关键避免过度拟合和增强对准性能。OSP 在两个广泛使用的人类偏好数据集中实现了最先进的对准性能，而且在有限的离线数据和跨领域任务推广方面，OSP 是效率高、稳健性高于主流的在线方法 RLHF 的选择。此外，具有自主偏好能力的 LLM 建立的 OSP 语言模型可以在无需外部监督的情况下高效自我改进。

在线自好的语言模型

Online Self-Preferring Language Models

We develop new voting mechanisms for the case when voters and candidates are
located in an arbitrary unknown metric space, and the goal is to choose a
candidate minimizing social cost: the total distance from the voters to this
candidate. Previous work has often assumed that only ordinal preferences of the
voters are known (instead of their true costs), and focused on minimizing
distortion: the quality of the chosen candidate as compared with the best
possible candidate. In this paper, we instead assume that a (very small) amount
of information is known about the voter preference strengths, not just about
their ordinal preferences. We provide mechanisms with much better distortion
when this extra information is known as compared to mechanisms which use only
ordinal information. We quantify tradeoffs between the amount of information
known about preference strengths and the achievable distortion. We further
provide advice about which type of information about preference strengths seems
to be the most useful. Finally, we conclude by quantifying the ideal candidate
distortion, which compares the quality of the chosen outcome with the best
possible candidate that could ever exist, instead of only the best candidate
that is actually in the running.

该研究开发了一种新的投票机制，旨在选择能够最大程度降低社会成本（即选民与候选人之间的总距离）的候选人，并量化了关于候选人失真度和投票者偏好力量信息之间的权衡。