Aligning large language models (LLMs) with human objectives is crucial for
real-world applications. However, fine-tuning LLMs for alignment often suffers
from unstable training and requires substantial computing resources. Test-time
alignment techniques, such as prompting and guided decoding, do not modify the
underlying model, and their performance remains dependent on the original
model's capabilities. To address these challenges, we propose aligning LLMs
through representation editing. The core of our method is to view a pre-trained
autoregressive LLM as a discrete-time stochastic dynamical system. To achieve
alignment for specific objectives, we introduce external control signals into
the state space of this language dynamical system. We train a value function
directly on the hidden states according to the Bellman equation, enabling
gradient-based optimization to obtain the optimal control signals at test time.
Our experiments demonstrate that our method outperforms existing test-time
alignment techniques while requiring significantly fewer resources compared to
fine-tuning methods.

通过表示编辑的方法，提出了一种通过在预训练的自回归大型语言模型上引入外部控制信号，以实现特定目标对齐的方法，实验证明该方法在测试时间上优于现有的测试时间对齐技术，并且相对于微调方法需要较少的资源。