When prompting a language model (LM), users frequently expect the model to
adhere to a set of behavioral principles across diverse tasks, such as
producing insightful content while avoiding harmful or biased language.
Instilling such principles into a model can be resource-intensive and
technically challenging, generally requiring human preference labels or
examples. We introduce SAMI, a method for teaching a pretrained LM to follow
behavioral principles that does not require any preference labels or
demonstrations. SAMI is an iterative algorithm that finetunes a pretrained LM
to increase the conditional mutual information between constitutions and
self-generated responses given queries from a datasest. On single-turn dialogue
and summarization, a SAMI-trained mistral-7b outperforms the initial pretrained
model, with win rates between 66% and 77%. Strikingly, it also surpasses an
instruction-finetuned baseline (mistral-7b-instruct) with win rates between 55%
and 57% on single-turn dialogue. SAMI requires a "principle writer" model; to
avoid dependence on stronger models, we further evaluate aligning a strong
pretrained model (mixtral-8x7b) using constitutions written by a weak
instruction-finetuned model (mistral-7b-instruct). The SAMI-trained
mixtral-8x7b outperforms both the initial model and the instruction-finetuned
model, achieving a 65% win rate on summarization. Our results indicate that a
pretrained LM can learn to follow constitutions without using preference
labels, demonstrations, or human oversight.

使用 SAMI 方法，将预训练的语言模型教导遵循行为原则，无需任何偏好标签或示范，结果显示模型在单轮对话和摘要任务上优于初始预训练模型和指导微调模型，且能够在不使用偏好标签、示范或人工监督的情况下学会遵循定义。

自我监督的互信息对齐：学习遵循原则而无需偏好标签

Self-Supervised Alignment with Mutual Information: Learning to Follow  Principles without Preference Labels

Constitutions are foundational legal documents that underpin the governmental
and societal structures. As such, they are a reflection of a nation's cultural
and social uniqueness, but also contribute to establish topics of universal
importance, like citizens' rights and duties (RD). In this work, using the
renowned GPT-3.5, we leverage generative large language models to understand
constitutional passages that transcend national boundaries. A key contribution
of our study is the introduction of a novel application of abstractive
summarization on a multi-source collection of constitutional texts, with a
focus on European countries' constitution passages related to RD topics. Our
results show the meaningfulness of GPT-3.5 to produce informative, coherent and
faithful summaries capturing RD topics across European countries.

运用 GPT-3.5 对多源宪法文本进行抽象概括，研究发现其能够准确捕捉欧洲国家的公民权益等相关主题。