Is vision good enough for language? Recent advancements in multimodal models
primarily stem from the powerful reasoning abilities of large language models
(LLMs). However, the visual component typically depends only on the
instance-level contrastive language-image pre-training (CLIP). Our research
reveals that the visual capabilities in recent multimodal LLMs (MLLMs) still
exhibit systematic shortcomings. To understand the roots of these errors, we
explore the gap between the visual embedding space of CLIP and vision-only
self-supervised learning. We identify ''CLIP-blind pairs'' - images that CLIP
perceives as similar despite their clear visual differences. With these pairs,
we construct the Multimodal Visual Patterns (MMVP) benchmark. MMVP exposes
areas where state-of-the-art systems, including GPT-4V, struggle with
straightforward questions across nine basic visual patterns, often providing
incorrect answers and hallucinated explanations. We further evaluate various
CLIP-based vision-and-language models and found a notable correlation between
visual patterns that challenge CLIP models and those problematic for multimodal
LLMs. As an initial effort to address these issues, we propose a Mixture of
Features (MoF) approach, demonstrating that integrating vision self-supervised
learning features with MLLMs can significantly enhance their visual grounding
capabilities. Together, our research suggests visual representation learning
remains an open challenge, and accurate visual grounding is crucial for future
successful multimodal systems.

通过对 CLIP 模型的视觉嵌入空间与仅视觉自监督学习的对比研究，我们发现最新的多模态大型语言模型（MLLMs）在视觉能力方面仍然存在系统性缺陷。为了解决这些问题，我们提出了一种特征混合（MoF）方法，通过将视觉自监督学习特征与 MLLMs 相结合，显著提高了它们的视觉基础能力，从而表明视觉表示学习仍然是一个待解决的问题，并且准确的视觉基础对于未来成功的多模态系统至关重要。