Large language models (LLMs) excel in natural language processing but demand
intensive computation. To mitigate this, various quantization methods have been
explored, yet they compromise LLM performance. This paper unveils a previously
overlooked type of outlier in LLMs. Such outliers are found to allocate most of
the attention scores on initial tokens of input, termed as pivot tokens, which
is crucial to the performance of quantized LLMs. Given that, we propose
IntactKV to generate the KV cache of pivot tokens losslessly from the
full-precision model. The approach is simple and easy to combine with existing
quantization solutions. Besides, IntactKV can be calibrated as additional LLM
parameters to boost the quantized LLMs further. Mathematical analysis also
proves that IntactKV effectively reduces the upper bound of quantization error.
Empirical results show that IntactKV brings consistent improvement and achieves
lossless weight-only INT4 quantization on various downstream tasks, leading to
the new state-of-the-art for LLM quantization.

该研究揭示了大语言模型中一种被忽视的异常值类型，这些异常值在输入的初始符号上分配了大部分的注意力得分，并且通过提出 IntactKV 的方法从完全精确的模型中无损生成了初始符号的 KV 缓存，进而提升了量化的大语言模型的性能。