Quantizing large language models (LLMs) presents significant challenges,
primarily due to outlier activations that compromise the efficiency of low-bit
representation. Traditional approaches mainly focus on solving Normal
Outliers-activations with consistently high magnitudes across all tokens.
However, these techniques falter when dealing with Massive Outliers, which are
significantly higher in value and often cause substantial performance losses
during low-bit quantization. In this study, we propose DuQuant, an innovative
quantization strategy employing rotation and permutation transformations to
more effectively eliminate both types of outliers. Initially, DuQuant
constructs rotation matrices informed by specific outlier dimensions,
redistributing these outliers across adjacent channels within different
rotation blocks. Subsequently, a zigzag permutation is applied to ensure a
balanced distribution of outliers among blocks, minimizing block-wise variance.
An additional rotation further enhances the smoothness of the activation
landscape, thereby improving model performance. DuQuant streamlines the
quantization process and demonstrates superior outlier management, achieving
top-tier results in multiple tasks with various LLM architectures even under
4-bit weight-activation quantization. Our code is available at
this https URL

本研究提出了一种创新的量化策略 ——DuQuant，采用旋转和置换变换更有效地消除异常激活，并在多个任务中表现出卓越的异常值管理能力，即使在 4 位权重 - 激活量化下也能取得顶级结果。

高级异常管理和低延迟闪存器件的高效旋转和置换

Rotation and Permutation for Advanced Outlier Management and Efficient  Quantization of LLMs

Transformer-based models have gained widespread popularity in both the
computer vision (CV) and natural language processing (NLP) fields. However,
significant challenges arise during post-training linear quantization, leading
to noticeable reductions in inference accuracy. Our study focuses on uncovering
the underlying causes of these accuracy drops and proposing a
quantization-friendly fine-tuning method, \textbf{QuantTune}. Firstly, our
analysis revealed that, on average, 65\% of quantization errors result from the
precision loss incurred by the dynamic range amplification effect of outliers
across the target Transformer-based models. Secondly, \textbf{QuantTune}
adjusts weights based on the deviation of outlier activations and effectively
constrains the dynamic ranges of the problematic activations. As a result, it
successfully mitigates the negative impact of outliers on the inference
accuracy of quantized models. Lastly, \textbf{QuantTune} can be seamlessly
integrated into the back-propagation pass in the fine-tuning process without
requiring extra complexity in inference software and hardware design. Our
approach showcases significant improvements in post-training quantization
across a range of Transformer-based models, including ViT, Bert-base, and OPT.
QuantTune reduces accuracy drops by 12.09\% at 8-bit quantization and 33.8\% at
7-bit compared to top calibration methods, outperforming state-of-the-art
solutions by over 18.84\% across ViT models.

我们的研究着重于发现 Transformer-based 模型在后训练线性量化过程中准确性下降的潜在原因，并提出了一种适用于量化的微调方法 QuantTune。该方法通过根据异常激活的偏差调整权重，有效控制了有问题激活的动态范围，从而在几种 Transformer-based 模型中实现了显著的后训练量化的改进。