Vision-Language Models (VLMs) combine visual perception with the general capabilities, such as reasoning, of Large Language Models (LLMs). However, the mechanisms by which these two abilities can be combined and contribute remain poorly understood. In this work, we explore to compose perception and reasoning through model merging that connects parameters of different models. Unlike previous works that often focus on merging models of the same kind, we propose merging models across modalities, enabling the incorporation of the reasoning capabilities of LLMs into VLMs. Through extensive experiments, we demonstrate that model merging offers a successful pathway to transfer reasoning abilities from LLMs to VLMs in a training-free manner. Moreover, we utilize the merged models to understand the internal mechanism of perception and reasoning and how merging affects it. We find that perception capabilities are predominantly encoded in the early layers of the model, whereas reasoning is largely facilitated by the middle-to-late layers. After merging, we observe that all layers begin to contribute to reasoning, whereas the distribution of perception abilities across layers remains largely unchanged. These observations shed light on the potential of model merging as a tool for multimodal integration and interpretation.

本研究针对视觉-语言模型（VLMs）与大型语言模型（LLMs）之间的感知与推理机制的不明确性，提出了通过跨模态模型融合的方法。研究发现，模型融合不仅成功地将LLMs的推理能力转移至VLMs，而且揭示了感知能力主要分布在模型的早期层，而推理能力则在中后期层得以促进，为多模态集成和解析提供了新的思路。

将推理引入视觉：通过模型融合理解感知和推理