Recent advancements in Large Language Models (LLMs) have facilitated the
development of Multimodal LLMs (MLLMs). Despite their impressive capabilities,
MLLMs often suffer from an over-reliance on unimodal biases (e.g., language
bias and vision bias), leading to incorrect answers in complex multimodal
tasks. To investigate this issue, we propose a causal framework to interpret
the biases in Visual Question Answering (VQA) problems. Within our framework,
we devise a causal graph to elucidate the predictions of MLLMs on VQA problems,
and assess the causal effect of biases through an in-depth causal analysis.
Motivated by the causal graph, we introduce a novel MORE dataset, consisting of
12,000 VQA instances. This dataset is designed to challenge MLLMs' abilities,
necessitating multi-hop reasoning and the surmounting of unimodal biases.
Furthermore, we propose two strategies to mitigate unimodal biases and enhance
MLLMs' reasoning capabilities, including a Decompose-Verify-Answer (DeVA)
framework for limited-access MLLMs and the refinement of open-source MLLMs
through fine-tuning. Extensive quantitative and qualitative experiments offer
valuable insights for future research.

近期，大型语言模型 (LLM) 的进展促进了多模态 LLM (MLLM) 的发展。虽然 MLLM 具有令人印象深刻的能力，但往往过于依赖单模态偏见（例如，语言偏见和视觉偏见），导致在复杂的多模态任务中回答错误。为了研究这个问题，我们提出了一个因果框架来解释视觉问答 (VQA) 问题中的偏见。在我们的框架中，我们设计了一个因果图来阐明 MLLMs 在 VQA 问题上的预测，并通过深入的因果分析评估偏见的因果效应。受到因果图的启发，我们引入了一个名为 MORE 的新数据集，包含了 12,000 个 VQA 实例。该数据集旨在挑战 MLLMs 的能力，需要进行多跳推理和克服单模态偏见。此外，我们提出了两种策略来减轻单模态偏见和增强 MLLMs 的推理能力，包括针对有限访问 MLLMs 的 “分解 - 验证 - 回答” (DeVA) 框架以及通过微调改进开源 MLLMs。广泛的定量和定性实验为未来的研究提供了宝贵的见解。