Multimodal learning methods with targeted unimodal learning objectives have
exhibited their superior efficacy in alleviating the imbalanced multimodal
learning problem. However, in this paper, we identify the previously ignored
gradient conflict between multimodal and unimodal learning objectives,
potentially misleading the unimodal encoder optimization. To well diminish
these conflicts, we observe the discrepancy between multimodal loss and
unimodal loss, where both gradient magnitude and covariance of the
easier-to-learn multimodal loss are smaller than the unimodal one. With this
property, we analyze Pareto integration under our multimodal scenario and
propose MMPareto algorithm, which could ensure a final gradient with direction
that is common to all learning objectives and enhanced magnitude to improve
generalization, providing innocent unimodal assistance. Finally, experiments
across multiple types of modalities and frameworks with dense cross-modal
interaction indicate our superior and extendable method performance. Our method
is also expected to facilitate multi-task cases with a clear discrepancy in
task difficulty, demonstrating its ideal scalability. The source code and
dataset are available at this https URL

多模态学习中存在之前被忽视的多模态与单模态学习目标之间的梯度冲突问题，本文提出了 MMPareto 算法来解决这一冲突，以提高泛化性能和单模态学习的精确度，实验证明了算法的优越性及可扩展性。

MMPareto：通过无害单模态辅助提升多模态学习

MMPareto: Boosting Multimodal Learning with Innocent Unimodal Assistance

In this work, we seek to predict camera poses across scenes with a multi-task
learning manner, where we view the localization of each scene as a new task. We
propose OFVL-MS, a unified framework that dispenses with the traditional
practice of training a model for each individual scene and relieves gradient
conflict induced by optimizing multiple scenes collectively, enabling efficient
storage yet precise visual localization for all scenes. Technically, in the
forward pass of OFVL-MS, we design a layer-adaptive sharing policy with a
learnable score for each layer to automatically determine whether the layer is
shared or not. Such sharing policy empowers us to acquire task-shared
parameters for a reduction of storage cost and task-specific parameters for
learning scene-related features to alleviate gradient conflict. In the backward
pass of OFVL-MS, we introduce a gradient normalization algorithm that
homogenizes the gradient magnitude of the task-shared parameters so that all
tasks converge at the same pace. Furthermore, a sparse penalty loss is applied
on the learnable scores to facilitate parameter sharing for all tasks without
performance degradation. We conduct comprehensive experiments on multiple
benchmarks and our new released indoor dataset LIVL, showing that OFVL-MS
families significantly outperform the state-of-the-arts with fewer parameters.
We also verify that OFVL-MS can generalize to a new scene with much few
parameters while gaining superior localization performance.

利用多任务学习方式预测场景中的相机姿态，提出了 OFVL-MS 统一框架，通过设计适应性共享策略和梯度归一化算法，在减少存储成本和解决梯度冲突的基础上提高了视觉定位性能。