Post-training quantization (PTQ) serves as a potent technique to accelerate
the inference of large language models (LLMs). Nonetheless, existing works
still necessitate a considerable number of floating-point (FP) operations
during inference, including additional quantization and de-quantization, as
well as non-linear operators such as RMSNorm and Softmax. This limitation
hinders the deployment of LLMs on the edge and cloud devices. In this paper, we
identify the primary obstacle to integer-only quantization for LLMs lies in the
large fluctuation of activations across channels and tokens in both linear and
non-linear operations. To address this issue, we propose I-LLM, a novel
integer-only fully-quantized PTQ framework tailored for LLMs. Specifically, (1)
we develop Fully-Smooth Block-Reconstruction (FSBR) to aggressively smooth
inter-channel variations of all activations and weights. (2) to alleviate
degradation caused by inter-token variations, we introduce a novel approach
called Dynamic Integer-only MatMul (DI-MatMul). This method enables dynamic
quantization in full-integer matrix multiplication by dynamically quantizing
the input and outputs with integer-only operations. (3) we design
DI-ClippedSoftmax, DI-Exp, and DI-Normalization, which utilize bit shift to
execute non-linear operators efficiently while maintaining accuracy. The
experiment shows that our I-LLM achieves comparable accuracy to the FP baseline
and outperforms non-integer quantization methods. For example, I-LLM can
operate at W4A4 with negligible loss of accuracy. To our knowledge, we are the
first to bridge the gap between integer-only quantization and LLMs. We've
published our code on anonymous.4open.science, aiming to contribute to the
advancement of this field.

该研究论文通过提出一种新型的整数化后训练量化框架 (I-LLM)，解决了大语言模型在部署边缘和云设备上仍需要大量浮点运算的问题。实验证明，I-LLM 在保持准确性的前提下，可以以 W4A4 进行操作，优于其他非整数量化方法。