Despite their great success across various multimodal tasks, Large
Vision-Language Models (LVLMs) are facing a prevalent problem with object
hallucinations, where the generated textual responses are inconsistent with
ground-truth objects in the given image. This paper investigates various LVLMs
and pinpoints attention deficiency toward discriminative local image features
as one root cause of object hallucinations. Specifically, LVLMs predominantly
attend to prompt-independent global image features, while failing to capture
prompt-relevant local features, consequently undermining the visual grounding
capacity of LVLMs and leading to hallucinations. To this end, we propose
Assembly of Global and Local Attention (AGLA), a training-free and
plug-and-play approach that mitigates object hallucinations by exploring an
ensemble of global features for response generation and local features for
visual discrimination simultaneously. Our approach exhibits an image-prompt
matching scheme that captures prompt-relevant local features from images,
leading to an augmented view of the input image where prompt-relevant content
is reserved while irrelevant distractions are masked. With the augmented view,
a calibrated decoding distribution can be derived by integrating generative
global features from the original image and discriminative local features from
the augmented image. Extensive experiments show that AGLA consistently
mitigates object hallucinations and enhances general perception capability for
LVLMs across various discriminative and generative benchmarks. Our code will be
released at this https URL

通过研究大型视觉语言模型（LVLMs），本论文确定了对象幻觉的一个根本原因是对辨别性局部图像特征的注意力不足。基于此，我们提出了全局和局部注意力组装（AGLA）方法，该方法通过同时探索用于响应生成的全局特征和用于视觉辨别的局部特征，从而减轻对象幻觉。大量实验证明，AGLA 方法能够一致地减轻对象幻觉并提高 LVLM 在各种辨别性和生成性基准任务中的感知能力。