Aligning large language models (LLMs) with human objectives is crucial for
real-world applications. However, fine-tuning LLMs for alignment often suffers
from unstable training and requires substantial computing resources. Test-time
alignment techniques, such as prompting and guided decoding, do not modify the
underlying model, and their performance remains dependent on the original
model's capabilities. To address these challenges, we propose aligning LLMs
through representation editing. The core of our method is to view a pre-trained
autoregressive LLM as a discrete-time stochastic dynamical system. To achieve
alignment for specific objectives, we introduce external control signals into
the state space of this language dynamical system. We train a value function
directly on the hidden states according to the Bellman equation, enabling
gradient-based optimization to obtain the optimal control signals at test time.
Our experiments demonstrate that our method outperforms existing test-time
alignment techniques while requiring significantly fewer resources compared to
fine-tuning methods.

通过表示编辑的方法，提出了一种通过在预训练的自回归大型语言模型上引入外部控制信号，以实现特定目标对齐的方法，实验证明该方法在测试时间上优于现有的测试时间对齐技术，并且相对于微调方法需要较少的资源。

用表示编辑来对齐大型语言模型：一个控制的观点

Aligning Large Language Models with Representation Editing: A Control  Perspective

Aligning pretrained language models (LMs) is a complex and resource-intensive
process, often requiring access to large amounts of ground-truth preference
data and substantial compute. Are these costs necessary? That is, it is
possible to align using only inherent model knowledge and without additional
training? We tackle this challenge with AlignEZ, a novel approach that uses (1)
self-generated preference data and (2) representation editing to provide nearly
cost-free alignment. During inference, AlignEZ modifies LM representations to
reduce undesirable and boost desirable components using subspaces identified
via self-generated preference pairs. Our experiments reveal that this nearly
cost-free procedure significantly narrows the gap between base pretrained and
tuned models by an average of 31.6%, observed across six datasets and three
model architectures. Additionally, we explore the potential of using AlignEZ as
a means of expediting more expensive alignment procedures. Our experiments show
that AlignEZ improves DPO models tuned only using a small subset of
ground-truth preference data. Lastly, we study the conditions under which
improvement using AlignEZ is feasible, providing valuable insights into its
effectiveness.

AlignEZ 是一种使用自动生成的偏好数据和表示编辑来几乎零成本对齐预训练语言模型的新方法，通过减少不必要且提升有利的组件，并利用自动生成的偏好对来识别子空间，这一几乎零成本的过程显著缩小了基本预训练模型和调整模型之间的差距（平均缩小 31.6%），并且可以用作加快更昂贵对齐过程的手段，同时提供了它的有效性条件下的有价值洞察。

能够免费自动对准吗？

Is Free Self-Alignment Possible?

Parameter Efficient Fine-Tuning (PEFT) has gained significant attention for
its ability to achieve competitive results while updating only a small subset
of trainable parameters. Despite the promising performance of current PEFT
methods, they present challenges in hyperparameter selection, such as
determining the rank of LoRA or Adapter, or specifying the length of soft
prompts. In addressing these challenges, we propose a novel approach to
fine-tuning neural models, termed Representation EDiting (RED), which scales
and biases the representation produced at each layer. RED substantially reduces
the number of trainable parameters by a factor of $25,700$ compared to full
parameter fine-tuning, and by a factor of $32$ compared to LoRA. Remarkably,
RED achieves comparable or superior results to full parameter fine-tuning and
other PEFT methods. Extensive experiments were conducted across models of
varying architectures and scales, including RoBERTa, GPT-2, T5, and Llama-2,
and the results demonstrate the efficiency and efficacy of RED, positioning it
as a promising PEFT approach for large neural models.

Parameter Efficient Fine-Tuning (PEFT) 方法和 Representation EDiting (RED) 方法是解决神经模型微调中的可调超参数选择和可训练参数数量减少的关键研究领域和主题，通过对多种模型进行广泛实验，证明了 RED 方法在大型神经模型中的高效性和优越性。