This paper explores a novel multi-modal alternating learning paradigm
pursuing a reconciliation between the exploitation of uni-modal features and
the exploration of cross-modal interactions. This is motivated by the fact that
current paradigms of multi-modal learning tend to explore multi-modal features
simultaneously. The resulting gradient prohibits further exploitation of the
features in the weak modality, leading to modality competition, where the
dominant modality overpowers the learning process. To address this issue, we
study the modality-alternating learning paradigm to achieve reconcilement.
Specifically, we propose a new method called ReconBoost to update a fixed
modality each time. Herein, the learning objective is dynamically adjusted with
a reconcilement regularization against competition with the historical models.
By choosing a KL-based reconcilement, we show that the proposed method
resembles Friedman's Gradient-Boosting (GB) algorithm, where the updated
learner can correct errors made by others and help enhance the overall
performance. The major difference with the classic GB is that we only preserve
the newest model for each modality to avoid overfitting caused by ensembling
strong learners. Furthermore, we propose a memory consolidation scheme and a
global rectification scheme to make this strategy more effective. Experiments
over six multi-modal benchmarks speak to the efficacy of the method. We release
the code at this https URL

研究介绍一种新的多模态交替学习范式，旨在平衡利用单模态特征和跨模态交互的探索，通过使用 ReconBoost 方法、调整学习目标、内存整合方案和全局矫正方案，实现多模态学习的有效性。