Preference datasets are essential for incorporating human preferences into
pre-trained language models, playing a key role in the success of Reinforcement
Learning from Human Feedback. However, these datasets often demonstrate
conflicting alignment objectives, leading to increased vulnerability to
jailbreak attacks and challenges in adapting downstream tasks to prioritize
specific alignment objectives without negatively impacting others. In this
work, we introduce a novel statistical metric, Alignment Dimension Conflict, to
quantify the degree of conflict within preference datasets. We then present
\texttt{Hummer} and its fine-grained variant, \texttt{Hummer-F}, as innovative
pairwise preference datasets with reduced-conflict alignment objectives.
\texttt{Hummer} is built based on UltraFeedback and is enhanced by AI feedback
from GPT-4, marking as the first preference dataset aimed at reducing the
competition between alignment objectives. Furthermore, we develop reward
models, HummerRM and HummerRM-F, which employ a hybrid sampling approach to
balance diverse alignment objectives effectively. This sampling method
positions HummerRM as an ideal model for domain-specific further fine-tuning
and reducing vulnerabilities to attacks.

引入了一种新的统计度量指标，即 Alignment Dimension Conflict，用于量化偏好数据集内部的冲突程度。提出了 Hummer 和 Hummer-F 这两个创新的成对偏好数据集，并开发了 HummerRM 和 HummerRM-F 这两个奖励模型，有效平衡多样的对齐目标，适用于领域特定的进一步微调和减少攻击的弱点。