Quantization stands as a pivotal technique for large language model (LLM) serving, yet it poses significant challenges particularly in achieving effective low-bit quantization. The limited numerical mapping makes the quantized model produce a non-trivial error, bringing out intolerable performance degration. This paper is anchored in the basic idea of model compression objectives, and delves into the layer-wise error distribution of LLMs during post-training quantization. Subsequently, we introduce ASER, an algorithm consisting of (1) Error Reconstruction: low-rank compensation for quantization error with LoRA-style matrices constructed by whitening SVD; (2) Activation Smoothing: outlier extraction to gain smooth activation and better error compensation. ASER is capable of quantizing typical LLMs to low-bit ones, particularly preserving accuracy even in W4A8 per-channel setup. Experimental results show that ASER is competitive among the state-of-the-art quantization algorithms, showing potential to activation quantization, with minor overhead.

本研究解决了大型语言模型（LLM）量化中有效低位量化的挑战，提出了一种新颖的ASER算法，通过强化量化后的误差重建和激活平滑来改善模型性能。实验结果表明，ASER在保持准确性的同时，能够将典型的LLM量化为低位模型，并在最先进的量化算法中表现出竞争力。

ASER：用于大型语言模型量化的激活平滑与误差重建