Although Large Visual Language Models (LVLMs) have demonstrated exceptional
abilities in understanding multimodal data, they invariably suffer from
hallucinations, leading to a disconnect between the generated text and the
corresponding images. Almost all current visual contrastive decoding methods
attempt to mitigate these hallucinations by introducing visual uncertainty
information that appropriately widens the contrastive logits gap between
hallucinatory and targeted ones. However, due to uncontrollable nature of the
global visual uncertainty, they struggle to precisely induce the hallucinatory
tokens, which severely limits their effectiveness in mitigating hallucinations
and may even lead to the generation of undesired hallucinations. To tackle this
issue, we conducted the theoretical analysis to promote the effectiveness of
contrast decoding. Building on this insight, we introduce a novel optimization
strategy named Hallucination-Induced Optimization (HIO). This strategy seeks to
amplify the contrast between hallucinatory and targeted tokens relying on a
fine-tuned theoretical preference model (i.e., Contrary Bradley-Terry Model),
thereby facilitating efficient contrast decoding to alleviate hallucinations in
LVLMs. Extensive experimental research demonstrates that our HIO strategy can
effectively reduce hallucinations in LVLMs, outperforming state-of-the-art
methods across various benchmarks.

利用 Contrary Bradley-Terry Model 进行优化，引入 Hallucination-Induced Optimization 策略，提高 Large Visual Language Models 处理视觉对比解码中的幻觉问题的效果。

通过诱导式优化减轻大型视觉语言模型中的幻觉

Alleviating Hallucinations in Large Vision-Language Models through  Hallucination-Induced Optimization

Large Vision-Language Models (LVLMs) have advanced considerably, intertwining
visual recognition and language understanding to generate content that is not
only coherent but also contextually attuned. Despite their success, LVLMs still
suffer from the issue of object hallucinations, where models generate plausible
yet incorrect outputs that include objects that do not exist in the images. To
mitigate this issue, we introduce Visual Contrastive Decoding (VCD), a simple
and training-free method that contrasts output distributions derived from
original and distorted visual inputs. The proposed VCD effectively reduces the
over-reliance on statistical bias and unimodal priors, two essential causes of
object hallucinations. This adjustment ensures the generated content is closely
grounded to visual inputs, resulting in contextually accurate outputs. Our
experiments show that VCD, without either additional training or the usage of
external tools, significantly mitigates the object hallucination issue across
different LVLM families. Beyond mitigating object hallucinations, VCD also
excels in general LVLM benchmarks, highlighting its wide-ranging applicability.

大视觉语言模型（LVLMs）通过视觉识别和语言理解相结合，生成连贯且与上下文相关的内容，但仍然存在物体幻觉问题。本文介绍一种名为 Visual Contrastive Decoding（VCD）的简单且无需训练的方法，通过对比原始和失真的视觉输入产生的输出分布，有效降低统计偏差和单模式先验产生物体幻觉的影响，确保生成的内容与视觉输入密切相关，从而产生上下文准确的输出。实验证明，VCD 不需要额外的训练或使用外部工具，在不同的 LVLM 族群中显著减轻了物体幻觉问题。除减轻物体幻觉问题外，VCD 在通用 LVLM 基准测试中也表现出色，展示了其广泛的适用性。