In the realm of Multimodal Large Language Models (MLLMs), vision-language
connector plays a crucial role to link the pre-trained vision encoders with
Large Language Models (LLMs). Despite its importance, the vision-language
connector has been relatively less explored. In this study, we aim to propose a
strong vision-language connector that enables MLLMs to achieve high accuracy
while maintain low computation cost. We first reveal the existence of the
visual anchors in Vision Transformer and propose a cost-effective search
algorithm to extract them. Building on these findings, we introduce the Anchor
Former (AcFormer), a novel vision-language connector designed to leverage the
rich prior knowledge obtained from these visual anchors during pretraining,
guiding the aggregation of information. Through extensive experimentation, we
demonstrate that the proposed method significantly reduces computational costs
by nearly two-thirds compared with baseline, while simultaneously outperforming
baseline methods. This highlights the effectiveness and efficiency of AcFormer.

本研究提出了一种强大的视觉 - 语言连接器，通过挖掘视觉锚点并在预训练中利用其丰富的先验知识，实现高准确性和低计算成本的多模态大型语言模型。通过广泛的实验验证，该方法相比基线方法将计算成本减少了三分之二，同时表现更好，突显了 AcFormer 的效果和效率。