Recent advancements in large multimodal models (LMMs) have significantly enhanced performance across diverse tasks, with ongoing efforts to further integrate additional modalities such as video and audio. However, most existing LMMs remain vulnerable to hallucinations, the discrepancy between the factual multimodal input and the generated textual output, which has limited their applicability in various real-world scenarios. This paper presents the first systematic investigation of hallucinations in LMMs involving the three most common modalities: language, visual, and audio. Our study reveals two key contributors to hallucinations: overreliance on unimodal priors and spurious inter-modality correlations. To address these challenges, we introduce the benchmark The Curse of Multi-Modalities (CMM), which comprehensively evaluates hallucinations in LMMs, providing a detailed analysis of their underlying issues. Our findings highlight key vulnerabilities, including imbalances in modality integration and biases from training data, underscoring the need for balanced cross-modal learning and enhanced hallucination mitigation strategies. Based on our observations and findings, we suggest potential research directions that could enhance the reliability of LMMs.

本研究系统性探讨了大型多模态模型（LMMs）在语言、视觉和音频模态中幻觉现象的问题，这一现象限制了其在现实场景中的应用。我们提出了“多模态的诅咒”（CMM）基准，深入分析幻觉产生的原因，并强调了需要在跨模态学习中实现平衡与增强幻觉缓解策略的重要性。

多模态的诅咒：评估大型多模态模型在语言、视觉和音频中的幻觉