Preference datasets are essential for incorporating human preferences into
pre-trained language models, playing a key role in the success of Reinforcement
Learning from Human Feedback. However, these datasets often demonstrate
conflicting alignment objectives, leading to increased vulnerability to
jailbreak attacks and challenges in adapting downstream tasks to prioritize
specific alignment objectives without negatively impacting others. In this
work, we introduce a novel statistical metric, Alignment Dimension Conflict, to
quantify the degree of conflict within preference datasets. We then present
\texttt{Hummer} and its fine-grained variant, \texttt{Hummer-F}, as innovative
pairwise preference datasets with reduced-conflict alignment objectives.
\texttt{Hummer} is built based on UltraFeedback and is enhanced by AI feedback
from GPT-4, marking as the first preference dataset aimed at reducing the
competition between alignment objectives. Furthermore, we develop reward
models, HummerRM and HummerRM-F, which employ a hybrid sampling approach to
balance diverse alignment objectives effectively. This sampling method
positions HummerRM as an ideal model for domain-specific further fine-tuning
and reducing vulnerabilities to attacks.

引入了一种新的统计度量指标，即 Alignment Dimension Conflict，用于量化偏好数据集内部的冲突程度。提出了 Hummer 和 Hummer-F 这两个创新的成对偏好数据集，并开发了 HummerRM 和 HummerRM-F 这两个奖励模型，有效平衡多样的对齐目标，适用于领域特定的进一步微调和减少攻击的弱点。

Hummer: 朝着有限竞争偏好数据集的方向

Hummer: Towards Limited Competitive Preference Dataset

Reports of human-like behaviors in foundation models are growing, with
psychological theories providing enduring tools to investigate these behaviors.
However, current research tends to directly apply these human-oriented tools
without verifying the faithfulness of their outcomes. In this paper, we
introduce a framework, RealBehavior, which is designed to characterize the
humanoid behaviors of models faithfully. Beyond simply measuring behaviors, our
framework assesses the faithfulness of results based on reproducibility,
internal and external consistency, and generalizability. Our findings suggest
that a simple application of psychological tools cannot faithfully characterize
all human-like behaviors. Moreover, we discuss the impacts of aligning models
with human and social values, arguing for the necessity of diversifying
alignment objectives to prevent the creation of models with restricted
characteristics.

通过引入 RealBehavior 框架，本文旨在忠实地表征模型的人类行为，评估结果的忠实性，并讨论与人类和社会价值观的模型对齐目标多样化的必要性。

RealBehavior: 一个准确描述基础模型人类行为机制的框架

RealBehavior: A Framework for Faithfully Characterizing Foundation  Models' Human-like Behavior Mechanisms

Previous cross-lingual transfer methods are restricted to orthographic
representation learning via textual scripts. This limitation hampers
cross-lingual transfer and is biased towards languages sharing similar
well-known scripts. To alleviate the gap between languages from different
writing scripts, we propose PhoneXL, a framework incorporating phonemic
transcriptions as an additional linguistic modality beyond the traditional
orthographic transcriptions for cross-lingual transfer. Particularly, we
propose unsupervised alignment objectives to capture (1) local one-to-one
alignment between the two different modalities, (2) alignment via
multi-modality contexts to leverage information from additional modalities, and
(3) alignment via multilingual contexts where additional bilingual dictionaries
are incorporated. We also release the first phonemic-orthographic alignment
dataset on two token-level tasks (Named Entity Recognition and Part-of-Speech
Tagging) among the understudied but interconnected
Chinese-Japanese-Korean-Vietnamese (CJKV) languages. Our pilot study reveals
phonemic transcription provides essential information beyond the orthography to
enhance cross-lingual transfer and bridge the gap among CJKV languages, leading
to consistent improvements on cross-lingual token-level tasks over
orthographic-based multilingual PLMs.

我们提出了 PhoneXL 框架，通过使用音素转录作为传统的正字转录之外的附加语言模态，以缓解不同书写脚本语言之间的差距，同时释放出首个音素 - 正字对齐数据集，并展示了音素转录提供了关键信息，能够增强 CJKV 语言之间的跨语言传输，从而在跨语言标记级任务上实现持续改进。