Recent advancements in Large Vision Language Models (LVLMs) have
revolutionized how machines understand and generate textual responses based on
visual inputs. Despite their impressive capabilities, they often produce
"hallucinatory" outputs that do not accurately reflect the visual information,
posing challenges in reliability and trustworthiness. Current methods such as
contrastive decoding have made strides in addressing these issues by
contrasting the original probability distribution of generated tokens with
distorted counterparts; yet, generating visually-faithful outputs remains a
challenge. In this work, we shift our focus to the opposite: What could serve
as a complementary enhancement to the original probability distribution? We
propose a simple, training-free method termed RITUAL to enhance robustness
against hallucinations in LVLMs. Our approach employs random image
transformations as complements to the original probability distribution, aiming
to mitigate the likelihood of hallucinatory visual explanations by enriching
the model's exposure to varied visual scenarios. Our empirical results show
that while the isolated use of transformed images initially degrades
performance, strategic implementation of these transformations can indeed serve
as effective complements. Notably, our method is compatible with current
contrastive decoding methods and does not require external models or costly
self-feedback mechanisms, making it a practical addition. In experiments,
RITUAL significantly outperforms existing contrastive decoding methods across
several object hallucination benchmarks, including POPE, CHAIR, and MME.

最近，大型视觉语言模型（LVLMs）的前沿发展彻底改变了机器基于视觉输入理解和生成文本响应的方式。然而，尽管这些模型的能力令人印象深刻，它们经常产生与视觉信息不符的 “幻觉性” 输出，给可靠性和可信度带来了挑战。当前的方法（如对比解码）在解决这些问题方面取得了一些进展，通过将生成的标记的原始概率分布与扭曲的对应物进行对比，然而生成视觉真实的输出仍然是一个挑战。在本研究中，我们将注意力转移到相反的方向上：什么能够作为原始概率分布的补充增强？我们提出了一种名为 RITUAL 的简单且无需训练的方法，以增强 LVLMs 对幻觉的鲁棒性。我们的方法利用随机图像变换作为原始概率分布的补充，旨在通过丰富模型对不同视觉情景的暴露来减少幻觉性视觉解释的可能性。我们的实证结果表明，尽管单独使用转换图像会最初降低性能，但有策略地实现这些转换确实可以作为有效的补充。值得注意的是，我们的方法与当前的对比解码方法兼容，并且不需要外部模型或昂贵的自反馈机制，因此具有实际应用价值。在实验中，RITUAL 在多个物体幻觉基准数据集（包括 POPE、CHAIR 和 MME）上显著优于现有的对比解码方法。