Multimodal learning helps to comprehensively understand the world, by
integrating different senses. Accordingly, multiple input modalities are
expected to boost model performance, but we actually find that they are not
fully exploited even when the multimodal model outperforms its uni-modal
counterpart. Specifically, in this paper we point out that existing multimodal
discriminative models, in which uniform objective is designed for all
modalities, could remain under-optimized uni-modal representations, caused by
another dominated modality in some scenarios, e.g., sound in blowing wind
event, vision in drawing picture event, etc. To alleviate this optimization
imbalance, we propose on-the-fly gradient modulation to adaptively control the
optimization of each modality, via monitoring the discrepancy of their
contribution towards the learning objective. Further, an extra Gaussian noise
that changes dynamically is introduced to avoid possible generalization drop
caused by gradient modulation. As a result, we achieve considerable improvement
over common fusion methods on different multimodal tasks, and this simple
strategy can also boost existing multimodal methods, which illustrates its
efficacy and versatility. The source code is available at
https://github.com/GeWu-Lab/OGM-GE_CVPR2022.

本文研究了多模式学习中可能存在的优化失衡问题，提出了一种新的梯度调节方法，通过动态监测不同输入模态的贡献来自适应地优化每个模态，以解决某些场景下存在的支配模态问题，并且在不同的多模式任务中能够获得显著的改进效果。

通过即时梯度调控实现平衡多模态学习

Balanced Multimodal Learning via On-the-fly Gradient Modulation

In many personalized recommendation scenarios, the generalization ability of
a target task can be improved via learning with additional auxiliary tasks
alongside this target task on a multi-task network. However, this method often
suffers from a serious optimization imbalance problem. On the one hand, one or
more auxiliary tasks might have a larger influence than the target task and
even dominate the network weights, resulting in worse recommendation accuracy
for the target task. On the other hand, the influence of one or more auxiliary
tasks might be too weak to assist the target task. More challenging is that
this imbalance dynamically changes throughout the training process and varies
across the parts of the same network. We propose a new method: MetaBalance to
balance auxiliary losses via directly manipulating their gradients w.r.t the
shared parameters in the multi-task network. Specifically, in each training
iteration and adaptively for each part of the network, the gradient of an
auxiliary loss is carefully reduced or enlarged to have a closer magnitude to
the gradient of the target loss, preventing auxiliary tasks from being so
strong that dominate the target task or too weak to help the target task.
Moreover, the proximity between the gradient magnitudes can be flexibly
adjusted to adapt MetaBalance to different scenarios. The experiments show that
our proposed method achieves a significant improvement of 8.34% in terms of
NDCG@10 upon the strongest baseline on two real-world datasets. The code of our
approach can be found at here: this https URL

本文提出了一种新方法 ——MetaBalance，通过直接操纵多任务网络中共享参数的梯度来平衡辅助任务的损失，以解决多任务学习时存在的优化失衡问题，实验结果表明该方法在两个真实数据集上相对于最强基线提高了 8.34％的 NDCG @ 10。