Large language models (LLMs) are still struggling in aligning with human
preference in complex tasks and scenarios. They are prone to overfit into the
unexpected patterns or superficial styles in the training data. We conduct an
empirical study that only selects the top-10\% most updated parameters in LLMs
for alignment training, and see improvements in the convergence process and
final performance. It indicates the existence of redundant neurons in LLMs for
alignment training. To reduce its influence, we propose a low-redundant
alignment method named \textbf{ALLO}, focusing on optimizing the most related
neurons with the most useful supervised signals. Concretely, we first identify
the neurons that are related to the human preference data by a gradient-based
strategy, then identify the alignment-related key tokens by reward models for
computing loss. Besides, we also decompose the alignment process into the
forgetting and learning stages, where we first forget the tokens with unaligned
knowledge and then learn aligned knowledge, by updating different ratios of
neurons, respectively. Experimental results on 10 datasets have shown the
effectiveness of ALLO. Our code and data are available at
https://github.com/RUCAIBox/ALLO.

在这篇研究论文中，研究人员通过对大型语言模型（LLMs）的经验研究发现了对齐训练中存在的冗余神经元，并提出了一种名为 ALLO 的低冗余对齐方法。该方法通过梯度策略识别与人类偏好数据相关的神经元，通过奖励模型计算损失来识别与对齐相关的关键词汇，并将对齐过程分解为遗忘和学习阶段，通过更新不同比例的神经元实现。实验证明 ALLO 的有效性。

大型语言模型对齐的低冗余优化

Low-Redundant Optimization for Large Language Model Alignment

Deep Neural nets (NNs) with millions of parameters are at the heart of many
state-of-the-art computer vision systems today. However, recent works have
shown that much smaller models can achieve similar levels of performance. In
this work, we address the problem of pruning parameters in a trained NN model.
Instead of removing individual weights one at a time as done in previous works,
we remove one neuron at a time. We show how similar neurons are redundant, and
propose a systematic way to remove them. Our experiments in pruning the densely
connected layers show that we can remove upto 85\% of the total parameters in
an MNIST-trained network, and about 35\% for AlexNet without significantly
affecting performance. Our method can be applied on top of most networks with a
fully connected layer to give a smaller network.

本研究探讨在已训练的神经网络模型中修剪神经元的问题并提出了一种系统性的方式去除冗余的相似神经元，并且通过在 MNIST 训练网络中修剪密集连接层，达到 85％ 的总参数去除率，在 AlexNet 上达到 35％去除率，并且没有明显影响其性能。