Currently, inspired by the success of vision-language models (VLMs), an increasing number of researchers are focusing on improving VLMs and have achieved promising results. However, most existing methods concentrate on optimizing the connector and enhancing the language model component, while neglecting improvements to the vision encoder itself. In contrast, we propose Text Guided LLaVA (TG-LLaVA) in this paper, which optimizes VLMs by guiding the vision encoder with text, offering a new and orthogonal optimization direction. Specifically, inspired by the purpose-driven logic inherent in human behavior, we use learnable latent embeddings as a bridge to analyze textual instruction and add the analysis results to the vision encoder as guidance, refining it. Subsequently, another set of latent embeddings extracts additional detailed text-guided information from high-resolution local patches as auxiliary information. Finally, with the guidance of text, the vision encoder can extract text-related features, similar to how humans focus on the most relevant parts of an image when considering a question. This results in generating better answers. Experiments on various datasets validate the effectiveness of the proposed method. Remarkably, without the need for additional training data, our propsoed method can bring more benefits to the baseline (LLaVA-1.5) compared with other concurrent methods. Furthermore, the proposed method consistently brings improvement in different settings.

本文针对现有视觉语言模型（VLMs）中视觉编码器提升不足的问题，提出了一种新的优化方向——文本引导的LLaVA（TG-LLaVA）。通过使用可学习的潜在嵌入，分析文本指令并将其作为指导输入到视觉编码器，显著改进了特征提取的相关性，实验结果表明该方法在不同设置下均优于现有技术。此研究无需额外训练数据，展现了可观的性能提升潜力。

文本引导的LLaVA：通过可学习潜在嵌入优化视觉语言模型