Making each modality in multi-modal data contribute is of vital importance to
learning a versatile multi-modal model. Existing methods, however, are often
dominated by one or few of modalities during model training, resulting in
sub-optimal performance. In this paper, we refer to this problem as modality
bias and attempt to study it in the context of multi-modal classification
systematically and comprehensively. After stepping into several empirical
analysis, we recognize that one modality affects the model prediction more just
because this modality has a spurious correlation with instance labels. In order
to primarily facilitate the evaluation on the modality bias problem, we
construct two datasets respectively for the colored digit recognition and video
action recognition tasks in line with the Out-of-Distribution (OoD) protocol.
Collaborating with the benchmarks in the visual question answering task, we
empirically justify the performance degradation of the existing methods on
these OoD datasets, which serves as evidence to justify the modality bias
learning. In addition, to overcome this problem, we propose a plug-and-play
loss function method, whereby the feature space for each label is adaptively
learned according to the training set statistics. Thereafter, we apply this
method on eight baselines in total to test its effectiveness. From the results
on four datasets regarding the above three tasks, our method yields remarkable
performance improvements compared with the baselines, demonstrating its
superiority on reducing the modality bias problem.

本文研究了在多模态分类系统中影响模型性能的模态偏差问题，通过构建两个基于 Out-of-Distribution 协议的数据集和提出一种自适应的 plug-and-play 损失函数方法，在彩色数字识别、视频动作识别和视觉问答三个任务上实现了明显的性能改进，证明了该方法在减少模态偏差问题方面的优越性。