Multimodal learning often outperforms its unimodal counterparts by exploiting
unimodal contributions and cross-modal interactions. However, focusing only on
integrating multimodal features into a unified comprehensive representation
overlooks the unimodal characteristics. In real data, the contributions of
modalities can vary from instance to instance, and they often reinforce or
conflict with each other. In this study, we introduce a novel \text{MultiModal}
loss paradigm for multimodal learning, which subgroups instances according to
their unimodal contributions. \text{MultiModal} loss can prevent inefficient
learning caused by overfitting and efficiently optimize multimodal models. On
synthetic data, \text{MultiModal} loss demonstrates improved classification
performance by subgrouping difficult instances within certain modalities. On
four real multimodal datasets, our loss is empirically shown to improve the
performance of recent models. Ablation studies verify the effectiveness of our
loss. Additionally, we show that our loss generates a reliable prediction score
for each modality, which is essential for subgrouping. Our \text{MultiModal}
loss is a novel loss function to subgroup instances according to the
contribution of modalities in multimodal learning and is applicable to a
variety of multimodal models with unimodal decisions. Our code is available at
this https URL

我们介绍了一种新的多模态学习损失范式，根据模态的贡献将实例分组，以防止过拟合引起的低效学习，并高效优化多模态模型，实验证明我们的损失提高了最近模型的性能。