Quantizing the activations of large language models (LLMs) has been a
significant challenge due to the presence of structured outliers. Most existing
methods focus on the per-token or per-tensor quantization of activations,
making it difficult to achieve both accuracy and hardware efficiency. To
address this problem, we propose OutlierTune, an efficient per-channel
post-training quantization (PTQ) method for the activations of LLMs.
OutlierTune consists of two components: pre-execution of dequantization and
symmetrization. The pre-execution of dequantization updates the model weights
by the activation scaling factors, avoiding the internal scaling and costly
additional computational overheads brought by the per-channel activation
quantization. The symmetrization further reduces the quantization differences
arising from the weight updates by ensuring the balanced numerical ranges
across different activation channels. OutlierTune is easy to implement and
hardware-efficient, introducing almost no additional computational overheads
during the inference. Extensive experiments show that the proposed framework
outperforms existing methods across multiple different tasks. Demonstrating
better generalization, this framework improves the Int6 quantization of the
instruction-tuning LLMs, such as OPT-IML, to the same level as half-precision
(FP16). Moreover, we have shown that the proposed framework is 1.48x faster
than the FP16 implementation while reducing approximately 2x memory usage.

OutlierTune 是一种针对 LLMs 激活的高效后训练量化方法，它通过预执行去量化和对称化两个组件，解决了大规模语言模型的激活量化准确性和硬件效率问题。