Long-context Multimodal Large Language Models (MLLMs) demand substantial
computational resources for inference as the growth of their multimodal
Key-Value (KV) cache, in response to increasing input lengths, challenges
memory and time efficiency. Unlike single-modality LLMs that manage only
textual contexts, the KV cache of long-context MLLMs includes representations
from multiple images with temporal and spatial relationships and related
textual contexts. The predominance of image tokens means traditional
optimizations for LLMs' KV caches are unsuitable for multimodal long-context
settings, and no prior works have addressed this challenge. In this work, we
introduce LOOK-M, a pioneering, fine-tuning-free approach that efficiently
reduces the multimodal KV cache size while maintaining performance comparable
to a full cache. We observe that during prompt prefill, the model prioritizes
more textual attention over image features, and based on the multimodal
interaction observation, a new proposed text-prior method is explored to
compress the KV cache. Furthermore, to mitigate the degradation of image
contextual information, we propose several compensatory strategies using KV
pairs merging. LOOK-M demonstrates that with a significant reduction in KV
Cache memory usage, such as reducing it by 80% in some cases, it not only
achieves up to 1.5x faster decoding but also maintains or even enhances
performance across a variety of long context multimodal tasks.

LOOK-M 是一个节省多模态 KV 缓存大小的创新方法，其通过优化文本和图像特征的交互作用，使用新的文本优先方法来压缩 KV 缓存，以及使用 KV 对的合并来缓解图像上下文信息的退化，实现了高效的解码速度和在各种多模态长上下文任务中保持或增强性能。