Multimodal learning methods with targeted unimodal learning objectives have
exhibited their superior efficacy in alleviating the imbalanced multimodal
learning problem. However, in this paper, we identify the previously ignored
gradient conflict between multimodal and unimodal learning objectives,
potentially misleading the unimodal encoder optimization. To well diminish
these conflicts, we observe the discrepancy between multimodal loss and
unimodal loss, where both gradient magnitude and covariance of the
easier-to-learn multimodal loss are smaller than the unimodal one. With this
property, we analyze Pareto integration under our multimodal scenario and
propose MMPareto algorithm, which could ensure a final gradient with direction
that is common to all learning objectives and enhanced magnitude to improve
generalization, providing innocent unimodal assistance. Finally, experiments
across multiple types of modalities and frameworks with dense cross-modal
interaction indicate our superior and extendable method performance. Our method
is also expected to facilitate multi-task cases with a clear discrepancy in
task difficulty, demonstrating its ideal scalability. The source code and
dataset are available at this https URL

多模态学习中存在之前被忽视的多模态与单模态学习目标之间的梯度冲突问题，本文提出了 MMPareto 算法来解决这一冲突，以提高泛化性能和单模态学习的精确度，实验证明了算法的优越性及可扩展性。