Multi-modal Large Language Models (MLLMs) demonstrate remarkable success
across various vision-language tasks. However, they suffer from visual
hallucination, where the generated responses diverge from the provided image.
Are MLLMs completely oblivious to accurate visual cues when they hallucinate?
Our investigation reveals that the visual branch may simultaneously advocate
both accurate and non-existent content. To address this issue, we propose
Pensieve, a training-free method inspired by our observation that analogous
visual hallucinations can arise among images sharing common semantic and
appearance characteristics. During inference, Pensieve enables MLLMs to
retrospect relevant images as references and compare them with the test image.
This paradigm assists MLLMs in downgrading hallucinatory content mistakenly
supported by the visual input. Experiments on Whoops, MME, POPE, and LLaVA
Bench demonstrate the efficacy of Pensieve in mitigating visual hallucination,
surpassing other advanced decoding strategies. Additionally, Pensieve aids
MLLMs in identifying details in the image and enhancing the specificity of
image descriptions.

通过提出 Pensieve 方法，研究发现多模态大语言模型在处理视觉幻觉时会同时支持准确和不存在的内容，Pensieve 方法通过在推理中回溯相关图像作为参考，并将其与测试图像进行比较，从而减少视觉幻觉现象，并提升图像描述的细节和特定性。