Recent advancements in the vision-language model have shown notable
generalization in vision-language tasks after visual instruction tuning.
However, bridging the gap between the pre-trained vision encoder and the large
language models becomes the whole network's bottleneck. To improve
cross-modality alignment, existing works usually consider more visual
instruction data covering a broader range of vision tasks to fine-tune the
model for question-answering, which are costly to obtain. However, the image
contains rich contextual information that has been largely under-explored. This
paper first attempts to harness this overlooked context within visual
instruction data, training the model to self-supervised `learning' how to ask
high-quality questions. In this way, we introduce a novel framework named
SQ-LLaVA: Self-Questioning for Large Vision-Language Assistant. SQ-LLaVA
exhibits proficiency in generating flexible and meaningful image-related
questions while analyzing the visual clue and prior language knowledge,
signifying an advanced level of generalized visual understanding. Moreover,
fine-tuning SQ-LLaVA on higher-quality instruction data shows a consistent
performance improvement compared with traditional visual-instruction tuning
methods. This improvement highlights the efficacy of self-questioning
techniques in achieving a deeper and more nuanced comprehension of visual
content across various contexts.

近期视觉语言模型的进展在视觉指导调整后，在视觉语言任务中展现了显著的泛化能力。这篇论文通过利用视觉指导数据中被忽视的上下文信息，训练模型进行自我监督学习以提问高质量问题，引入了称为 SQ-LLaVA 的新框架。SQ-LLaVA 在分析视觉线索和先前的语言知识时表现出高水准的泛化视觉理解能力，与传统的视觉指导调整方法相比，将 SQ-LLaVA 在更高质量的指导数据上进行微调可以持续提高性能，突显了自问技术在不同语境下实现更深入和细腻的视觉内容理解能力。