There is growing consensus that language model (LM) developers should not be
the sole deciders of LM behavior, creating a need for methods that enable the
broader public to collectively shape the behavior of LM systems that affect
them. To address this need, we present Collective Constitutional AI (CCAI): a
multi-stage process for sourcing and integrating public input into LMs-from
identifying a target population to sourcing principles to training and
evaluating a model. We demonstrate the real-world practicality of this approach
by creating what is, to our knowledge, the first LM fine-tuned with
collectively sourced public input and evaluating this model against a baseline
model trained with established principles from a LM developer. Our quantitative
evaluations demonstrate several benefits of our approach: the CCAI-trained
model shows lower bias across nine social dimensions compared to the baseline
model, while maintaining equivalent performance on language, math, and
helpful-harmless evaluations. Qualitative comparisons of the models suggest
that the models differ on the basis of their respective constitutions, e.g.,
when prompted with contentious topics, the CCAI-trained model tends to generate
responses that reframe the matter positively instead of a refusal. These
results demonstrate a promising, tractable pathway toward publicly informed
development of language models.

使用 Collective Constitutional AI 方法，通过公众参与培训和评估，创建了第一个经集体数据微调的语言模型，与基线模型相比，在九个社会维度上展现出更低的偏见，同时在语言、数学和有益 - 无害评估上表现相当的性能。

集体宪政 AI：将语言模型与公众输入对齐

Collective Constitutional AI: Aligning a Language Model with Public  Input

Language Models (LMs) often must integrate facts they memorized in
pretraining with new information that appears in a given context. These two
sources can disagree, causing competition within the model, and it is unclear
how an LM will resolve the conflict. On a dataset that queries for knowledge of
world capitals, we investigate both distributional and mechanistic determinants
of LM behavior in such situations. Specifically, we measure the proportion of
the time an LM will use a counterfactual prefix (e.g., "The capital of Poland
is London") to overwrite what it learned in pretraining ("Warsaw"). On Pythia
and GPT2, the training frequency of both the query country ("Poland") and the
in-context city ("London") highly affect the models' likelihood of using the
counterfactual. We then use head attribution to identify individual attention
heads that either promote the memorized answer or the in-context answer in the
logits. By scaling up or down the value vector of these heads, we can control
the likelihood of using the in-context answer on new data. This method can
increase the rate of generating the in-context answer to 88\% of the time
simply by scaling a single head at runtime. Our work contributes to a body of
evidence showing that we can often localize model behaviors to specific
components and provides a proof of concept for how future methods might control
model behavior dynamically at runtime.

利用头部归因方法控制模型行为，将模型表现局部化，并提供了一种概念验证方法，展示如何在运行时动态地控制模型行为。