Large Vision-Language Models (LVLMs) are pivotal for real-world AI tasks like embodied intelligence due to their strong vision-language reasoning abilities. However, current LVLMs process entire images at the token level, which is inefficient compared to humans who analyze information and generate content at the conceptual level, extracting relevant visual concepts with minimal effort. This inefficiency, stemming from the lack of a visual concept model, limits LVLMs' usability in real-world applications. To address this, we propose VCM, an end-to-end self-supervised visual concept modeling framework. VCM leverages implicit contrastive learning across multiple sampled instances and vision-language fine-tuning to construct a visual concept model without requiring costly concept-level annotations. Our results show that VCM significantly reduces computational costs (e.g., 85\% fewer FLOPs for LLaVA-1.5-7B) while maintaining strong performance across diverse image understanding tasks. Moreover, VCM enhances visual encoders' capabilities in classic visual concept perception tasks. Extensive quantitative and qualitative experiments validate the effectiveness and efficiency of VCM.

本研究针对当前大型视觉-语言模型在处理整个图像时效率低下的问题，提出了一种端到端的自监督视觉概念建模框架VCM。该方法通过隐式对比学习和视觉-语言微调构建视觉概念模型，显著降低计算成本，同时保持在各种图像理解任务中的优良性能，从而提升了视觉编码器在经典视觉概念感知任务中的能力。

基于隐式对比学习的视觉概念建模：视觉语言指令微调