Recently, multimodal large language models have exploded with an endless variety, most of the popular Large Vision Language Models (LVLMs) depend on sequential visual representation, where images are converted into hundreds or thousands of tokens before being input into the Large Language Model (LLM) along with language prompts. The black-box design hinders the interpretability of visual-language models, especially regarding more complex reasoning tasks. To explore the interaction process between image and text in complex reasoning tasks, we introduce the information flow method to visualize the interaction mechanism. By analyzing the dynamic flow of the information flow, we find that the information flow appears to converge in the shallow layer. Further investigation revealed a redundancy of the image token in the shallow layer. Consequently, a truncation strategy was introduced to aggregate image tokens within these shallow layers. This approach has been validated through experiments across multiple models, yielding consistent improvements.

通过分析信息流的动态流动，我们发现信息流似乎在浅层中汇聚，进一步的研究揭示了浅层中图像令牌的冗余，因此引入了一个截断策略来聚合这些浅层内的图像令牌，该方法通过多个模型上的实验证实，并获得了一致的改进。

从冗余到相关性：增强多模态大型语言模型的可解释性