Existing multimodal tasks mostly target at the complete input modality setting, i.e., each modality is either complete or completely missing in both training and test sets. However, the randomly missing situations have still been underexplored. In this paper, we present a novel approach named MM-Align to address the missing-modality inference problem. Concretely, we propose 1) an alignment dynamics learning module based on the theory of optimal transport (OT) for indirect missing data imputation; 2) a denoising training algorithm to simultaneously enhance the imputation results and backbone network performance. Compared with previous methods which devote to reconstructing the missing inputs, MM-Align learns to capture and imitate the alignment dynamics between modality sequences. Results of comprehensive experiments on three datasets covering two multimodal tasks empirically demonstrate that our method can perform more accurate and faster inference and relieve overfitting under various missing conditions.

本文介绍一种名为MM-Align的新方法，利用最优传输理论中的对齐动力学学习模块和去噪训练算法，针对缺失多模态数据进行推断。通过在覆盖两个多模态任务的三个数据集上进行全面实验，结果表明我们的方法可以在各种缺失条件下执行更精确和更快的推断，并减轻过度拟合问题。

MM-Align: 基于最优输运的对齐动力学学习，以实现快速准确推断缺失模态序列