Long-context Multimodal Large Language Models (MLLMs) demand substantial
computational resources for inference as the growth of their multimodal
Key-Value (KV) cache, in response to increasing input lengths, challenges
memory and time efficiency. Unlike single-modality LLMs that manage only
textual contexts, the KV cache of long-context MLLMs includes representations
from multiple images with temporal and spatial relationships and related
textual contexts. The predominance of image tokens means traditional
optimizations for LLMs' KV caches are unsuitable for multimodal long-context
settings, and no prior works have addressed this challenge. In this work, we
introduce LOOK-M, a pioneering, fine-tuning-free approach that efficiently
reduces the multimodal KV cache size while maintaining performance comparable
to a full cache. We observe that during prompt prefill, the model prioritizes
more textual attention over image features, and based on the multimodal
interaction observation, a new proposed text-prior method is explored to
compress the KV cache. Furthermore, to mitigate the degradation of image
contextual information, we propose several compensatory strategies using KV
pairs merging. LOOK-M demonstrates that with a significant reduction in KV
Cache memory usage, such as reducing it by 80% in some cases, it not only
achieves up to 1.5x faster decoding but also maintains or even enhances
performance across a variety of long context multimodal tasks.

LOOK-M 是一个节省多模态 KV 缓存大小的创新方法，其通过优化文本和图像特征的交互作用，使用新的文本优先方法来压缩 KV 缓存，以及使用 KV 对的合并来缓解图像上下文信息的退化，实现了高效的解码速度和在各种多模态长上下文任务中保持或增强性能。

LOOK-M: KV 缓存中的一次查找优化，用于高效的多模态长上下文推理

LOOK-M: Look-Once Optimization in KV Cache for Efficient Multimodal  Long-Context Inference

Serving LLMs requires substantial memory due to the storage requirements of
Key-Value (KV) embeddings in the KV cache, which grows with sequence length. An
effective approach to compress KV cache is quantization. However, traditional
quantization methods face significant memory overhead due to the need to store
quantization constants (at least a zero point and a scale) in full precision
per data block. Depending on the block size, this overhead can add 1 or 2 bits
per quantized number. We introduce QJL, a new quantization approach that
consists of a Johnson-Lindenstrauss (JL) transform followed by sign-bit
quantization. In contrast to existing methods, QJL eliminates memory overheads
by removing the need for storing quantization constants. We propose an
asymmetric estimator for the inner product of two vectors and demonstrate that
applying QJL to one vector and a standard JL transform without quantization to
the other provides an unbiased estimator with minimal distortion. We have
developed an efficient implementation of the QJL sketch and its corresponding
inner product estimator, incorporating a lightweight CUDA kernel for optimized
computation. When applied across various LLMs and NLP tasks to quantize the KV
cache to only 3 bits, QJL demonstrates a more than fivefold reduction in KV
cache memory usage without compromising accuracy, all while achieving faster
runtime. Codes are available at https://github.com/amirzandieh/QJL.

通过使用 QJL 方法，我们成功实现了对 LLMs 的服务，将 KV 缓存量化至 3 位，缩减了缓存的内存使用量并提高了运行速度，而同时保持了准确性。