Large Multimodal Models (LMMs) have shown significant reasoning capabilities
by connecting a visual encoder and a large language model. LMMs typically use a
fixed amount of visual tokens, such as the penultimate layer features in the
CLIP visual encoder, as the prefix content. Recent LMMs incorporate more
complex visual inputs, such as high-resolution images and videos, which
increase the number of visual tokens significantly. However, due to the design
of the Transformer architecture, computational costs associated with these
models tend to increase quadratically with the number of input tokens. To
tackle this problem, we explore a token reduction mechanism and find, similar
to prior work, that many visual tokens are spatially redundant. Based on this,
we propose PruMerge, a novel adaptive visual token reduction approach, which
largely reduces the number of visual tokens while maintaining comparable model
performance. We first select the unpruned visual tokens based on their
similarity to class tokens and spatial tokens. We then cluster the pruned
tokens based on key similarity and merge the clustered tokens with the unpruned
tokens to supplement their information. Empirically, when applied to LLaVA-1.5,
our approach can compress the visual tokens by 14.4 times on average, and
achieve comparable performance across diverse visual question-answering and
reasoning tasks. Code and checkpoints are at this https URL

通过减少视觉标记并合并相关标记，我们提出了一种自适应的视觉标记压缩方法 PruMerge，可以显著减少可视标记的数量并保持相似的模型性能。