Multimodal Large Language Models (LLMs) have recently achieved promising
zero-shot accuracy on visual question answering (VQA) -- a fundamental task
affecting various downstream applications and domains. Given the great
potential for the broad use of these models, it is important to investigate
their limitations in dealing with different image and question properties. In
this work, we investigate whether multimodal LLMs can perceive small details as
well as large details in images. In particular, we show that their zero-shot
accuracy in answering visual questions is very sensitive to the size of the
visual subject of the question, declining up to $46\%$ with size. Furthermore,
we show that this effect is causal by observing that human visual cropping can
significantly mitigate their sensitivity to size. Inspired by the usefulness of
human cropping, we then propose three automatic visual cropping methods as
inference time mechanisms to improve the zero-shot performance of multimodal
LLMs. We study their effectiveness on four popular VQA datasets, and a subset
of the VQAv2 dataset tailored towards fine visual details. Our findings suggest
that multimodal LLMs should be used with caution in detail-sensitive VQA
applications, and that visual cropping is a promising direction to improve
their zero-shot performance. Our code and data are publicly available.

这项研究探讨了多模态大型语言模型在处理不同图像和问题属性时是否能够感知图像的细节，并表明它们在回答视觉问题的零样本准确性受到问题中视觉主体大小的敏感性，但通过人工裁剪可以显著缓解这种敏感性影响，并提出了三种自动图像裁剪方法用于改进多模态大型语言模型的零样本性能。

视觉裁剪提升多模态大型语言模型的零指导问题回答

Visual Cropping Improves Zero-Shot Question Answering of Multimodal  Large Language Models

Visual Question Answering is a challenging task, as it requires seamless
interaction between perceptual, linguistic, and background knowledge systems.
While the recent progress of visual and natural language models like BLIP has
led to improved performance on this task, we lack understanding of the ability
of such models to perform on different kinds of questions and reasoning types.
As our initial analysis of BLIP-family models revealed difficulty with
answering fine-detail questions, we investigate the following question: Can
visual cropping be employed to improve the performance of state-of-the-art
visual question answering models on fine-detail questions? Given the recent
success of the BLIP-family models, we study a zero-shot and a fine-tuned BLIP
model. We define three controlled subsets of the popular VQA-v2 benchmark to
measure whether cropping can help model performance. Besides human cropping, we
devise two automatic cropping strategies based on multi-modal embedding by CLIP
and BLIP visual QA model gradients. Our experiments demonstrate that the
performance of BLIP model variants can be significantly improved through human
cropping, and automatic cropping methods can produce comparable benefits. A
deeper dive into our findings indicates that the performance enhancement is
more pronounced in zero-shot models than in fine-tuned models and more salient
with smaller bounding boxes than larger ones. We perform case studies to
connect quantitative differences with qualitative observations across question
types and datasets. Finally, we see that the cropping enhancement is robust, as
we gain an improvement of 4.59% (absolute) in the general VQA-random task by
simply inputting a concatenation of the original and gradient-based cropped
images. We make our code available to facilitate further innovation on visual
cropping methods for question answering.

文章探讨将视觉切割技术用于改进最新的 BLIP 模型在细节问题上的表现，提出了三种测试方式，并进行了实验验证，结果表明，相较于人工切割，基于多模式嵌入的自动切割方法效果相似，同时发现对于零 - shot 模型和小边界框，效果更为显著，在 VQA-random 任务中获得了 4.59% 的改善。