As deep learning advances, there is an ever-growing demand for models capable
of synthesizing information from multi-modal resources to address the complex
tasks raised from real-life applications. Recently, many large multi-modal
datasets have been collected, on which researchers actively explore different
methods of fusing multi-modal information. However, little attention has been
paid to quantifying the contribution of different modalities within the
proposed models. In this paper, we propose the {\bf SH}apley v{\bf A}lue-based
{\bf PE}rceptual (SHAPE) scores that measure the marginal contribution of
individual modalities and the degree of cooperation across modalities. Using
these scores, we systematically evaluate different fusion methods on different
multi-modal datasets for different tasks. Our experiments suggest that for some
tasks where different modalities are complementary, the multi-modal models
still tend to use the dominant modality alone and ignore the cooperation across
modalities. On the other hand, models learn to exploit cross-modal cooperation
when different modalities are indispensable for the task. In this case, the
scores indicate it is better to fuse different modalities at relatively early
stages. We hope our scores can help improve the understanding of how the
present multi-modal models operate on different modalities and encourage more
sophisticated methods of integrating multiple modalities.

本文主要介绍了一种基于 SHAPLEY 值的 PErceptual（SHAPE）得分，该得分可以测量单个模态的边际贡献和跨模态合作程度以及如何在不同任务的不同多模态数据集上系统地评估不同的融合方法，最终结果表明对于一些任务，在不同模态互补的情况下，多模态模型仍倾向于仅使用主导模态并忽略跨模态合作，而在不同模态必须挑战任务的情况下，模型则可以学习利用跨模态合作。