Reliably controlling the behavior of large language models (LLMs) is a
pressing open problem. Existing methods include supervised finetuning,
reinforcement learning from human feedback (RLHF), prompt engineering and
guided decoding. We instead investigate activation engineering: modifying
activations at inference time to predictably alter model behavior. In
particular, we bias the forward pass with an added 'steering vector' implicitly
specified through natural language.
Unlike past work which learned these steering vectors (Subramani, Suresh, and
Peters 2022; Hernandez, Li, and Andreas 2023), our Activation Addition (ActAdd)
method computes them by taking the activation differences that result from
pairs of prompts. We demonstrate ActAdd on GPT-2 on OpenWebText and ConceptNet.
Our inference-time approach yields control over high-level properties of output
and preserves off-target model performance. It involves far less compute and
implementation effort compared to finetuning or RLHF, allows users to provide
natural language specifications, and its overhead scales naturally with model
size.

控制大型语言模型行为的问题已成为紧迫的开放问题。在本文中，我们提出了一种称为 Activation Addition (ActAdd) 的方法，通过在推理过程中修改激活来可预测地改变模型行为，并展示了其在 GPT-2 上的应用，以及其与微调或强化学习从人类反馈中得到控制的方法相比所需的计算量和实施工作的差异。