Multimodal Large Language Models (LLMs) have recently achieved promising
zero-shot accuracy on visual question answering (VQA) -- a fundamental task
affecting various downstream applications and domains. Given the great
potential for the broad use of these models, it is important to investigate
their limitations in dealing with different image and question properties. In
this work, we investigate whether multimodal LLMs can perceive small details as
well as large details in images. In particular, we show that their zero-shot
accuracy in answering visual questions is very sensitive to the size of the
visual subject of the question, declining up to $46\%$ with size. Furthermore,
we show that this effect is causal by observing that human visual cropping can
significantly mitigate their sensitivity to size. Inspired by the usefulness of
human cropping, we then propose three automatic visual cropping methods as
inference time mechanisms to improve the zero-shot performance of multimodal
LLMs. We study their effectiveness on four popular VQA datasets, and a subset
of the VQAv2 dataset tailored towards fine visual details. Our findings suggest
that multimodal LLMs should be used with caution in detail-sensitive VQA
applications, and that visual cropping is a promising direction to improve
their zero-shot performance. Our code and data are publicly available.

这项研究探讨了多模态大型语言模型在处理不同图像和问题属性时是否能够感知图像的细节，并表明它们在回答视觉问题的零样本准确性受到问题中视觉主体大小的敏感性，但通过人工裁剪可以显著缓解这种敏感性影响，并提出了三种自动图像裁剪方法用于改进多模态大型语言模型的零样本性能。