The key-value (KV) cache accelerates LLMs decoding by storing KV tensors from previously generated tokens. It reduces redundant computation at the cost of increased memory usage. To mitigate this overhead, existing approaches compress KV tensors into lower-bit representations; however, quantization errors can accumulate as more tokens are generated, potentially resulting in undesired outputs. In this paper, we introduce SQuat (Subspace-orthogonal KV cache quantization). It first constructs a subspace spanned by query tensors to capture the most critical task-related information. During key tensor quantization, it enforces that the difference between the (de)quantized and original keys remains orthogonal to this subspace, minimizing the impact of quantization errors on the attention mechanism's outputs. SQuat requires no model fine-tuning, no additional calibration dataset for offline learning, and is grounded in a theoretical framework we develop. Through numerical experiments, we show that our method reduces peak memory by 2.17 to 2.82, improves throughput by 2.45 to 3.60, and achieves more favorable benchmark scores than existing KV cache quantization algorithms.

本文针对现有键值缓存量化方法中的量化误差累积问题提出了SQuat方法，该方法通过构建一个由查询张量生成的子空间来捕捉关键任务相关信息。在量化过程中，它确保量化后的键与原始键之间的差异与子空间正交，从而减小量化误差对注意力机制输出的影响。实验结果表明，SQuat在减少内存占用和提升处理能力方面表现显著优于现有算法。

子空间正交的键值缓存量化