Multimodal learning methods with targeted unimodal learning objectives have
exhibited their superior efficacy in alleviating the imbalanced multimodal
learning problem. However, in this paper, we identify the previously ignored
gradient conflict between multimodal and unimodal learning objectives,
potentially misleading the unimodal encoder optimization. To well diminish
these conflicts, we observe the discrepancy between multimodal loss and
unimodal loss, where both gradient magnitude and covariance of the
easier-to-learn multimodal loss are smaller than the unimodal one. With this
property, we analyze Pareto integration under our multimodal scenario and
propose MMPareto algorithm, which could ensure a final gradient with direction
that is common to all learning objectives and enhanced magnitude to improve
generalization, providing innocent unimodal assistance. Finally, experiments
across multiple types of modalities and frameworks with dense cross-modal
interaction indicate our superior and extendable method performance. Our method
is also expected to facilitate multi-task cases with a clear discrepancy in
task difficulty, demonstrating its ideal scalability. The source code and
dataset are available at this https URL

多模态学习中存在之前被忽视的多模态与单模态学习目标之间的梯度冲突问题，本文提出了 MMPareto 算法来解决这一冲突，以提高泛化性能和单模态学习的精确度，实验证明了算法的优越性及可扩展性。

MMPareto：通过无害单模态辅助提升多模态学习

MMPareto: Boosting Multimodal Learning with Innocent Unimodal Assistance

In multimodal machine learning, multiple modalities of data (e.g., text and
images) are combined to facilitate the learning of a better machine learning
model, which remains applicable to a corresponding unimodal task (e.g., text
generation). Recently, multimodal machine learning has enjoyed huge empirical
success (e.g. GPT-4). Motivated to develop theoretical justification for this
empirical success, Lu (NeurIPS '23, ALT '24) introduces a theory of multimodal
learning, and considers possible separations between theoretical models of
multimodal and unimodal learning. In particular, Lu (ALT '24) shows a
computational separation, which is relevant to worst-case instances of the
learning task.
In this paper, we give a stronger average-case computational separation,
where for "typical" instances of the learning task, unimodal learning is
computationally hard, but multimodal learning is easy. We then question how
"organic" the average-case separation is. Would it be encountered in practice?
To this end, we prove that under natural conditions, any given computational
separation between average-case unimodal and multimodal learning tasks implies
a corresponding cryptographic key agreement protocol. We suggest to interpret
this as evidence that very strong computational advantages of multimodal
learning may arise infrequently in practice, since they exist only for the
"pathological" case of inherently cryptographic distributions. However, this
does not apply to possible (super-polynomial) statistical advantages.

多模式机器学习与单模式学习的理论模型、计算分离以及平均情况的计算分离是本篇论文的主要研究内容。