Multimodal learning with incomplete input data (missing modality) is
practical and challenging. In this work, we conduct an in-depth analysis of
this challenge and find that modality dominance has a significant negative
impact on the model training, greatly degrading the missing modality
performance. Motivated by Grad-CAM, we introduce a novel indicator, gradients,
to monitor and reduce modality dominance which widely exists in the
missing-modality scenario. In aid of this indicator, we present a novel
Gradient-guided Modality Decoupling (GMD) method to decouple the dependency on
dominating modalities. Specifically, GMD removes the conflicted gradient
components from different modalities to achieve this decoupling, significantly
improving the performance. In addition, to flexibly handle modal-incomplete
data, we design a parameter-efficient Dynamic Sharing (DS) framework which can
adaptively switch on/off the network parameters based on whether one modality
is available. We conduct extensive experiments on three popular multimodal
benchmarks, including BraTS 2018 for medical segmentation, CMU-MOSI, and
CMU-MOSEI for sentiment analysis. The results show that our method can
significantly outperform the competitors, showing the effectiveness of the
proposed solutions. Our code is released here:
this https URL

本文介绍了一种使用渐变指示器和梯度引导方法来解决多模态学习中的缺失模态和模态主导性的问题，同时设计了一种动态共享框架来处理模态不完整的数据，并在三个多模态基准数据集上进行了实验证明了该方法的有效性。