The alignment problem in Large Language Models (LLMs) involves adapting them
to the broad spectrum of human values. This requirement challenges existing
alignment methods due to diversity of preferences and regulatory standards.
This paper introduces a novel alignment paradigm, priority rule following,
which defines rules as the primary control mechanism in each dialog,
prioritizing them over user instructions. Our preliminary analysis reveals that
even the advanced LLMs, such as GPT-4, exhibit shortcomings in understanding
and prioritizing the rules. Therefore, we present PriorityDistill, a
semi-automated approach for distilling priority following signals from LLM
simulations to ensure robust rule integration and adherence. Our experiments
show that this method not only effectively minimizes misalignments utilizing
only one general rule but also adapts smoothly to various unseen rules,
ensuring they are shielded from hijacking and that the model responds
appropriately.

该论文引入了一种新的对齐范式，即优先规则遵循，通过将规则定义为每个对话中的主要控制机制，并将其置于用户指令之上，来解决大语言模型（LLMs）中的对齐问题。作者提出了 PriorityDistill 方法，通过从 LLM 模拟中提取优先遵循信号，确保规则的稳健整合和遵守。实验证明，该方法不仅有效地最小化了错误对齐，而且能够平稳适应各种未见规则，确保其免受劫持并获得适当的模型响应。