Modern large language models (LLMs) have established state-of-the-art
performance through architectural improvements, but still require significant
computational cost for inference. In an effort to reduce the inference cost,
post-training quantization (PTQ) has become a popular approach, quantizing
weights and activations to lower precision, such as INT8. In this paper, we
reveal the challenges of activation quantization in GLU variants, which are
widely used in feed-forward network (FFN) of modern LLMs, such as LLaMA family.
The problem is that severe local quantization errors, caused by excessive
magnitudes of activation in GLU variants, significantly degrade the performance
of the quantized LLM. We denote these activations as activation spikes. Our
further observations provide a systematic pattern of activation spikes: 1) The
activation spikes occur in the FFN of specific layers, particularly in the
early and late layers, 2) The activation spikes are dedicated to a couple of
tokens, rather than being shared across a sequence. Based on our observations,
we propose two empirical methods, Quantization-free Module (QFeM) and
Quantization-free Prefix (QFeP), to isolate the activation spikes during
quantization. Our extensive experiments validate the effectiveness of the
proposed methods for the activation quantization, especially with
coarse-grained scheme, of latest LLMs with GLU variants, including LLaMA-2/3,
Mistral, Mixtral, SOLAR, and Gemma. In particular, our methods enhance the
current alleviation techniques (e.g., SmoothQuant) that fail to control the
activation spikes. Code is available at
this https URL

现代大型语言模型通过架构改进取得了最先进的性能，但仍需要昂贵的计算成本进行推理。本文研究了 GLU 变体中激活量化的挑战，揭示了过量激活量级引起的严重局部量化误差，提出了两种经验方法来隔离激活峰值，并验证了在最新的 GLU 变体的大型语言模型中的有效性。