In the realm of vision-language understanding, the proficiency of models in
interpreting and reasoning over visual content has become a cornerstone for
numerous applications. However, it is challenging for the visual encoder in
Large Vision-Language Models (LVLMs) to extract useful features tailored to
questions that aid the language model's response. Furthermore, a common
practice among existing LVLMs is to utilize lower-resolution images, which
restricts the ability for visual recognition. Our work introduces the
Chain-of-Spot (CoS) method, which we describe as Interactive Reasoning, a novel
approach that enhances feature extraction by focusing on key regions of
interest (ROI) within the image, corresponding to the posed questions or
instructions. This technique allows LVLMs to access more detailed visual
information without altering the original image resolution, thereby offering
multi-granularity image features. By integrating Chain-of-Spot with
instruct-following LLaVA-1.5 models, the process of image reasoning
consistently improves performance across a wide range of multimodal datasets
and benchmarks without bells and whistles and achieves new state-of-the-art
results. Our empirical findings demonstrate a significant improvement in LVLMs'
ability to understand and reason about visual content, paving the way for more
sophisticated visual instruction-following applications. Code and models are
available at this https URL

介绍了一种名为 Chain-of-Spot（CoS）的交互推理方法，通过关注图像中与问题或指令相对应的关键区域（ROI），增强特征提取，提供了多粒度图像特征，从而显著提高了大型视觉 - 语言模型（LVLMs）在理解和推理视觉内容方面的能力。