As the key component in multimodal large language models (MLLMs), the ability
of the visual encoder greatly affects MLLM's understanding on diverse image
content. Although some large-scale pretrained vision encoders such as vision
encoders in CLIP and DINOv2 have brought promising performance, we found that
there is still no single vision encoder that can dominate various image content
understanding, e.g., the CLIP vision encoder leads to outstanding results on
general image understanding but poor performance on document or chart content.
To alleviate the bias of CLIP vision encoder, we first delve into the inherent
behavior of different pre-trained vision encoders and then propose the MoVA, a
powerful and novel MLLM, adaptively routing and fusing task-specific vision
experts with a coarse-to-fine mechanism. In the coarse-grained stage, we design
a context-aware expert routing strategy to dynamically select the most suitable
vision experts according to the user instruction, input image, and expertise of
vision experts. This benefits from the powerful model function understanding
ability of the large language model (LLM) equipped with expert-routing low-rank
adaptation (LoRA). In the fine-grained stage, we elaborately conduct the
mixture-of-vision-expert adapter (MoV-Adapter) to extract and fuse
task-specific knowledge from various experts. This coarse-to-fine paradigm
effectively leverages representations from experts based on multimodal context
and model expertise, further enhancing the generalization ability. We conduct
extensive experiments to evaluate the effectiveness of the proposed approach.
Without any bells and whistles, MoVA can achieve significant performance gains
over current state-of-the-art methods in a wide range of challenging multimodal
benchmarks. Codes and models will be available at
this https URL

通过将任务特定视觉专家与粗到细的机制自适应地路由和融合，MoVA 可在各种具有挑战性的多模态基准测试中显著提高性能。