LLMs are seeing growing use for applications such as document analysis and
summarization which require large context windows, and with these large context
windows KV cache activations surface as the dominant contributor to memory
consumption during inference. Quantization is a promising approach for
compressing KV cache activations; however, existing solutions fail to represent
activations accurately in ultra-low precisions, such as sub-4-bit. In this
work, we present KVQuant, which addresses this problem by incorporating novel
methods for quantizing cached KV activations, including: (i) Per-Channel Key
Quantization, where we adjust the dimension along which we quantize the Key
activations to better match the distribution; (ii) Pre-RoPE Key Quantization,
where we quantize Key activations before the rotary positional embedding to
mitigate its impact on quantization; (iii) Non-Uniform KV Cache Quantization,
where we derive per-layer sensitivity-weighted non-uniform datatypes that
better represent the distributions; (iv) Per-Vector Dense-and-Sparse
Quantization, where we isolate outliers separately for each vector to minimize
skews in quantization ranges; and (v) Q-Norm, where we normalize quantization
centroids in order to mitigate distribution shift, providing additional
benefits for 2-bit quantization. By applying our method to the LLaMA, LLaMA-2,
and Mistral models, we achieve $<0.1$ perplexity degradation with 3-bit
quantization on both Wikitext-2 and C4, outperforming existing approaches. Our
method enables serving the LLaMA-7B model with a context length of up to 1
million on a single A100-80GB GPU and up to 10 million on an 8-GPU system.

LLMs 在大文本分析和摘要等需要大上下文窗口的应用中得到越来越广泛的应用，KV 缓存激活成为推断过程中存储器占用的主要贡献者。本研究通过引入新颖的方法对缓存的 KV 激活进行量化，包括：（i）通道关键激活量化，（ii）Rotary 位置嵌入本地化量化，（iii）非均匀 KV 缓存量化，（iv）向量稠密稀疏量化，和（v）Q-Norm。通过将这些方法应用于 LLaMA、LLaMA-2 和 Mistral 模型，我们在 Wikitext-2 和 C4 数据集上使用 3 位量化实现了小于 0.1 的困惑度退化，优于现有方法。我们的方法使得在单个 A100-80GB GPU 上可以为 LLaMA-7B 模型提供长达 100 万的上下文长度，而在 8-GPU 系统上可以提供长达 1000 万的上下文长度。