KV cache stores key and value states from previous tokens to avoid
re-computation, yet it demands substantial storage space, especially for long
sequences. Adaptive KV cache compression seeks to discern the saliency of
tokens, preserving vital information while aggressively compressing those of
less importance. However, previous methods of this approach exhibit significant
performance degradation at high compression ratios due to inaccuracies in
identifying salient tokens. In this paper, we present ZipCache, an accurate and
efficient KV cache quantization method for LLMs. First, we construct a strong
baseline for quantizing KV cache. Through the proposed channel-separable
tokenwise quantization scheme, the memory overhead of quantization parameters
are substantially reduced compared to fine-grained groupwise quantization. To
enhance the compression ratio, we propose normalized attention score as an
effective metric for identifying salient tokens by considering the lower
triangle characteristics of the attention matrix. Moreover, we develop an
efficient approximation method that decouples the saliency metric from full
attention scores, enabling compatibility with fast attention implementations
like FlashAttention. Extensive experiments demonstrate that ZipCache achieves
superior compression ratios, fast generation speed and minimal performance
losses compared with previous KV cache compression methods. For instance, when
evaluating Mistral-7B model on GSM8k dataset, ZipCache is capable of
compressing the KV cache by $4.98\times$, with only a $0.38\%$ drop in
accuracy. In terms of efficiency, ZipCache also showcases a $37.3\%$ reduction
in prefill-phase latency, a $56.9\%$ reduction in decoding-phase latency, and a
$19.8\%$ reduction in GPU memory usage when evaluating LLaMA3-8B model with a
input length of $4096$.

ZipCache 是一种用于 LLMs 的准确而高效的 KV 缓存量化方法，通过通道可分离的逐标记量化方案可显著减少量化参数的内存开销，并通过考虑注意力矩阵的下三角特性提出了规范化注意力分数作为识别重要标记的有效度量，以获得更高的压缩比，同时提供了快速生成速度和最小性能损失。