As the open community of large language models (LLMs) matures, multimodal LLMs (MLLMs) have promised an elegant bridge between vision and language. However, current research is inherently constrained by challenges such as the need for high-quality instruction pairs and the loss of visual information in image-to-text training objectives. To this end, we propose a Visual Token Complement framework (VTC) that helps MLLMs regain the missing visual features and thus improve response accuracy. Specifically, our VTC integrates text-to-image generation as a guide to identifying the text-irrelevant features, and a visual selector is then developed to generate complementary visual tokens to enrich the original visual input. Moreover, an iterative strategy is further designed to extract more visual information by iteratively using the visual selector without any additional training. Notably, the training pipeline requires no additional image-text pairs, resulting in a desired instruction tuning-free property. Both qualitative and quantitative experiments demonstrate the superiority and efficiency of our VTC.

本研究针对现有多模态大型语言模型（MLLMs）在训练中面临的高质量指令对需求和视觉信息丢失的问题，提出了一种视觉令牌补充框架（VTC）。该框架利用文本生成图像来识别与文本无关的特征，并通过视觉选择器生成补充的视觉令牌，从而增强原始视觉输入，显著提高响应准确性，且无需额外的图像-文本对，具备无指令调优的特性。

无指令调优的视觉令牌补充框架用于多模态大型语言模型