Large language models (LLMs) are still struggling in aligning with human
preference in complex tasks and scenarios. They are prone to overfit into the
unexpected patterns or superficial styles in the training data. We conduct an
empirical study that only selects the top-10\% most updated parameters in LLMs
for alignment training, and see improvements in the convergence process and
final performance. It indicates the existence of redundant neurons in LLMs for
alignment training. To reduce its influence, we propose a low-redundant
alignment method named \textbf{ALLO}, focusing on optimizing the most related
neurons with the most useful supervised signals. Concretely, we first identify
the neurons that are related to the human preference data by a gradient-based
strategy, then identify the alignment-related key tokens by reward models for
computing loss. Besides, we also decompose the alignment process into the
forgetting and learning stages, where we first forget the tokens with unaligned
knowledge and then learn aligned knowledge, by updating different ratios of
neurons, respectively. Experimental results on 10 datasets have shown the
effectiveness of ALLO. Our code and data are available at
https://github.com/RUCAIBox/ALLO.

在这篇研究论文中，研究人员通过对大型语言模型（LLMs）的经验研究发现了对齐训练中存在的冗余神经元，并提出了一种名为 ALLO 的低冗余对齐方法。该方法通过梯度策略识别与人类偏好数据相关的神经元，通过奖励模型计算损失来识别与对齐相关的关键词汇，并将对齐过程分解为遗忘和学习阶段，通过更新不同比例的神经元实现。实验证明 ALLO 的有效性。

大型语言模型对齐的低冗余优化

Low-Redundant Optimization for Large Language Model Alignment

Special-purpose hardware accelerators are increasingly pivotal for sustaining
performance improvements in emerging applications, especially as the benefits
of technology scaling continue to diminish. However, designers currently lack
effective tools and methodologies to construct complex, high-performance
accelerator architectures in a productive manner. Existing high-level synthesis
(HLS) tools often require intrusive source-level changes to attain satisfactory
quality of results. Despite the introduction of several new accelerator design
languages (ADLs) aiming to enhance or replace HLS, their advantages are more
evident in relatively simple applications with a single kernel. Existing ADLs
prove less effective for realistic hierarchical designs with multiple kernels,
even if the design hierarchy is flattened.
In this paper, we introduce Allo, a composable programming model for
efficient spatial accelerator design. Allo decouples hardware customizations,
including compute, memory, communication, and data type from algorithm
specification, and encapsulates them as a set of customization primitives. Allo
preserves the hierarchical structure of an input program by combining
customizations from different functions in a bottom-up, type-safe manner. This
approach facilitates holistic optimizations that span across function
boundaries. We conduct comprehensive experiments on commonly-used HLS
benchmarks and several realistic deep learning models. Our evaluation shows
that Allo can outperform state-of-the-art HLS tools and ADLs on all test cases
in the PolyBench. For the GPT2 model, the inference latency of the Allo
generated accelerator is 1.7x faster than the NVIDIA A100 GPU with 5.4x higher
energy efficiency, demonstrating the capability of Allo to handle large-scale
designs.

通过使用 Allo 编程模型，我们提出了一种有效的空间加速器设计方法，能够在各种应用和深度学习模型中取得更好的性能和能源效率，相比于 NVIDIA A100 GPU，Allo 生成的加速器在 GPT2 模型上具有 1.7 倍的推理延迟和 5.4 倍的能源效率提升。