Spurious bias, a tendency to use spurious correlations between non-essential
input attributes and target variables for predictions, has revealed a severe
robustness pitfall in deep learning models trained on single modality data.
Multimodal Large Language Models (MLLMs), which integrate both vision and
language models, have demonstrated strong capability in joint vision-language
understanding. However, whether spurious biases are prevalent in MLLMs remains
under-explored. We mitigate this gap by analyzing the spurious biases in a
multimodal setting, uncovering the specific test data patterns that can
manifest this problem when biases in the vision model cascade into the
alignment between visual and text tokens in MLLMs. To better understand this
problem, we introduce MM-SpuBench, a comprehensive visual question-answering
(VQA) benchmark designed to evaluate MLLMs' reliance on nine distinct
categories of spurious correlations from five open-source image datasets. The
VQA dataset is built from human-understandable concept information
(attributes). Leveraging this benchmark, we conduct a thorough evaluation of
current state-of-the-art MLLMs. Our findings illuminate the persistence of the
reliance on spurious correlations from these models and underscore the urge for
new methodologies to mitigate spurious biases. To support the MLLM robustness
research, we release our VQA benchmark at
this https URL

在深度学习模型中，单一模态数据的训练容易导致假的偏见，而多模态大型语言模型（MLLMs）在综合视觉和语言模型方面展示了强大的能力。本文分析了 MLLMs 中的假偏见，揭示了当视觉模型中的偏见影响 MLLMs 中视觉和文本符号之间的对齐时，特定的测试数据模式会表现出这一问题，并通过引入 MM-SpuBench、一个全面的视觉问答（VQA）评估基准，从五个开源图像数据集中评估了现有最先进的 MLLMs。我们的研究结果显示了这些模型对于假关联的依赖性的持久存在，并强调了减轻假的偏见的新方法的迫切性。为了支持 MLLMs 的稳健性研究，我们在该网址发布了我们的 VQA 基准。