Multimodal learning helps to comprehensively understand the world, by
integrating different senses. Accordingly, multiple input modalities are
expected to boost model performance, but we actually find that they are not
fully exploited even when the multimodal model outperforms its uni-modal
counterpart. Specifically, in this paper we point out that existing multimodal
discriminative models, in which uniform objective is designed for all
modalities, could remain under-optimized uni-modal representations, caused by
another dominated modality in some scenarios, e.g., sound in blowing wind
event, vision in drawing picture event, etc. To alleviate this optimization
imbalance, we propose on-the-fly gradient modulation to adaptively control the
optimization of each modality, via monitoring the discrepancy of their
contribution towards the learning objective. Further, an extra Gaussian noise
that changes dynamically is introduced to avoid possible generalization drop
caused by gradient modulation. As a result, we achieve considerable improvement
over common fusion methods on different multimodal tasks, and this simple
strategy can also boost existing multimodal methods, which illustrates its
efficacy and versatility. The source code is available at
https://github.com/GeWu-Lab/OGM-GE_CVPR2022.

本文研究了多模式学习中可能存在的优化失衡问题，提出了一种新的梯度调节方法，通过动态监测不同输入模态的贡献来自适应地优化每个模态，以解决某些场景下存在的支配模态问题，并且在不同的多模式任务中能够获得显著的改进效果。