How to efficiently serve LLMs in practice has become exceptionally challenging due to their prohibitive memory and computation requirements. In this study, we investigate optimizing the KV cache, whose memory footprint poses a critical bottleneck in LLM inference, especially when dealing with long context tasks. To tackle the challenge, we introduce MiniKV, a KV cache optimization method that simultaneously preserves long context task accuracy while significantly reducing KV cache size via a novel 2-bit layer-discriminative KV cache. More importantly, we develop specialized CUDA kernels to make MiniKV compatible with FlashAttention. Experiments on a wide range of long context tasks show that MiniKV effectively achieves 86% KV cache compression ratio while recovering over 98.5% of accuracy, outperforming state-of-the-art methods while achieving excellent measured system performance improvements.

本研究针对大语言模型（LLM）推理中的KV缓存占用内存问题进行优化，提出了MiniKV方法。该方法通过一种新颖的2位层区分KV缓存技术，在显著减少缓存大小的同时，保持了长上下文任务的高准确性。实验表明，MiniKV实现了86%的缓存压缩率，同时准确性恢复率超过98.5%，性能优于现有方法。

MiniKV：通过2位层区分KV缓存突破LLM推理的极限