In the realm of vision-language understanding, the proficiency of models in
interpreting and reasoning over visual content has become a cornerstone for
numerous applications. However, it is challenging for the visual encoder in
Large Vision-Language Models (LVLMs) to extract useful features tailored to
questions that aid the language model's response. Furthermore, a common
practice among existing LVLMs is to utilize lower-resolution images, which
restricts the ability for visual recognition. Our work introduces the
Chain-of-Spot (CoS) method, which we describe as Interactive Reasoning, a novel
approach that enhances feature extraction by focusing on key regions of
interest (ROI) within the image, corresponding to the posed questions or
instructions. This technique allows LVLMs to access more detailed visual
information without altering the original image resolution, thereby offering
multi-granularity image features. By integrating Chain-of-Spot with
instruct-following LLaVA-1.5 models, the process of image reasoning
consistently improves performance across a wide range of multimodal datasets
and benchmarks without bells and whistles and achieves new state-of-the-art
results. Our empirical findings demonstrate a significant improvement in LVLMs'
ability to understand and reason about visual content, paving the way for more
sophisticated visual instruction-following applications. Code and models are
available at this https URL

介绍了一种名为 Chain-of-Spot（CoS）的交互推理方法，通过关注图像中与问题或指令相对应的关键区域（ROI），增强特征提取，提供了多粒度图像特征，从而显著提高了大型视觉 - 语言模型（LVLMs）在理解和推理视觉内容方面的能力。

链式侦测：交互推理提升大型视觉语言模型

Chain-of-Spot: Interactive Reasoning Improves Large Vision-Language  Models

Images are often more significant than only the pixels to human eyes, as we
can infer, associate, and reason with contextual information from other sources
to establish a more complete picture. For example, in Figure 1, we can find a
way to identify the news articles related to the picture through segment-wise
understandings of the signs, the buildings, the crowds, and more. This
reasoning could provide the time and place the image was taken, which will help
us in subsequent tasks, such as automatic storyline construction, correction of
image source in intended effect photographs, and upper-stream processing such
as image clustering for certain location or time.
In this work, we formulate this problem and introduce TARA: a dataset with
16k images with their associated news, time, and location, automatically
extracted from New York Times, and an additional 61k examples as distant
supervision from WIT. On top of the extractions, we present a crowdsourced
subset in which we believe it is possible to find the images' spatio-temporal
information for evaluation purpose. We show that there exists a $70\%$ gap
between a state-of-the-art joint model and human performance, which is slightly
filled by our proposed model that uses segment-wise reasoning, motivating
higher-level vision-language joint models that can conduct open-ended reasoning
with world knowledge. The data and code are publicly available at
this https URL.

本研究介绍了 TARA 数据集，并探索了基于分段推理的开放式推理问题，结果发现现有模型和人类表现之间存在 70% 的差距。