Vision-Language Models (VLMs) are powerful tools for processing and understanding text and images. We study the processing of visual tokens in the language model component of LLaVA, a prominent VLM. Our approach focuses on analyzing the localization of object information, the evolution of visual token representations across layers, and the mechanism of integrating visual information for predictions. Through ablation studies, we demonstrated that object identification accuracy drops by over 70\% when object-specific tokens are removed. We observed that visual token representations become increasingly interpretable in the vocabulary space across layers, suggesting an alignment with textual tokens corresponding to image content. Finally, we found that the model extracts object information from these refined representations at the last token position for prediction, mirroring the process in text-only language models for factual association tasks. These findings provide crucial insights into how VLMs process and integrate visual information, bridging the gap between our understanding of language and vision models, and paving the way for more interpretable and controllable multimodal systems.

本研究针对视觉语言模型（VLMs）在处理图像和文本时的视觉信息处理进行了深入探索，揭示了对象信息的定位、视觉标记表示的演变及其集成机制。研究发现，删除对象特定标记会导致识别准确率下降超过70%，并表明视觉标记在层次上与文本标记的对应关系逐步增强，这对理解语言和视觉模型的整合提供了重要见解，推动了多模态系统的可解释性与可控性的发展。

朝着理解视觉信息处理在视觉语言模型中的应用