Modeling expressive cross-modal interactions seems crucial in multimodal
tasks, such as visual question answering. However, sometimes high-performing
black-box algorithms turn out to be mostly exploiting unimodal signals in the
data. We propose a new diagnostic tool, empirical multimodally-additive
function projection (EMAP), for isolating whether or not cross-modal
interactions improve performance for a given model on a given task. This
function projection modifies model predictions so that cross-modal interactions
are eliminated, isolating the additive, unimodal structure. For seven
image+text classification tasks (on each of which we set new state-of-the-art
benchmarks), we find that, in many cases, removing cross-modal interactions
results in little to no performance degradation. Surprisingly, this holds even
when expressive models, with capacity to consider interactions, otherwise
outperform less expressive models; thus, performance improvements, even when
present, often cannot be attributed to consideration of cross-modal feature
interactions. We hence recommend that researchers in multimodal machine
learning report the performance not only of unimodal baselines, but also the
EMAP of their best-performing model.

提出一种名为 EMAP 的诊断工具，用于检测跨模态交互是否提高给定模型在给定任务上的性能，在七个图像 + 文本分类任务中发现，即使具有考虑交互作用的表达模型在表现上优于不太表达的模型，删除跨模态交互也往往不会降低性能。建议多模式机器学习研究人员不仅报告单模基线的性能，而且还要报告最佳性能模型的 EMAP。