Key-Value (KV) Caching has become an essential technique for accelerating the
inference speed and throughput of generative Large Language Models~(LLMs).
However, the memory footprint of the KV cache poses a critical bottleneck in
LLM deployment as the cache size grows with batch size and sequence length,
often surpassing even the size of the model itself. Although recent methods
were proposed to select and evict unimportant KV pairs from the cache to reduce
memory consumption, the potential ramifications of eviction on the generative
process are yet to be thoroughly examined. In this paper, we examine the
detrimental impact of cache eviction and observe that unforeseen risks arise as
the information contained in the KV pairs is exhaustively discarded, resulting
in safety breaches, hallucinations, and context loss. Surprisingly, we find
that preserving even a small amount of information contained in the evicted KV
pairs via reduced precision quantization substantially recovers the incurred
degradation. On the other hand, we observe that the important KV pairs must be
kept at a relatively higher precision to safeguard the generation quality.
Motivated by these observations, we propose \textit{Mixed-precision KV
cache}~(MiKV), a reliable cache compression method that simultaneously
preserves the context details by retaining the evicted KV pairs in
low-precision and ensure generation quality by keeping the important KV pairs
in high-precision. Experiments on diverse benchmarks and LLM backbones show
that our proposed method offers a state-of-the-art trade-off between
compression ratio and performance, compared to other baselines.

通过实验证明，即使通过减少精度量化保存被清除的 KV 对中的一小部分信息，也能大大恢复引起的退化；同时，对于重要的 KV 对，必须保留较高的精度以确保生成质量。基于这些观察，我们提出了一种可靠的缓存压缩方法，即混合精度 KV 缓存（MiKV），通过将被清除的 KV 对保留在低精度以保留上下文细节，并通过将重要的 KV 对保留在高精度来确保生成质量。与其他基准相比，我们的方法在多种基准和 LLM 骨干上提供了一个最先进的压缩比和性能的权衡。