While the field of multi-modal learning keeps growing fast, the deficiency of
the standard joint training paradigm has become clear through recent studies.
They attribute the sub-optimal performance of the jointly trained model to the
modality competition phenomenon. Existing works attempt to improve the jointly
trained model by modulating the training process. Despite their effectiveness,
those methods can only apply to late fusion models. More importantly, the
mechanism of the modality competition remains unexplored. In this paper, we
first propose an adaptive gradient modulation method that can boost the
performance of multi-modal models with various fusion strategies. Extensive
experiments show that our method surpasses all existing modulation methods.
Furthermore, to have a quantitative understanding of the modality competition
and the mechanism behind the effectiveness of our modulation method, we
introduce a novel metric to measure the competition strength. This metric is
built on the mono-modal concept, a function that is designed to represent the
competition-less state of a modality. Through systematic investigation, our
results confirm the intuition that the modulation encourages the model to rely
on the more informative modality. In addition, we find that the jointly trained
model typically has a preferred modality on which the competition is weaker
than other modalities. However, this preferred modality need not dominate
others. Our code will be available at
this https URL

通过引入自适应梯度调制方法，本文提出了一种能够提升多模态模型性能的方法，并通过实验证明其优于现有的调制方法。此外，引入一种新的度量指标用于衡量模态竞争的强度，并系统研究了调制机制鼓励模型依赖更具信息量的模态。