Multimodal learning aims to imitate human beings to acquire complementary
information from multiple modalities for various downstream tasks. However,
traditional aggregation-based multimodal fusion methods ignore the
inter-modality relationship, treat each modality equally, suffer sensor noise,
and thus reduce multimodal learning performance. In this work, we propose a
novel multimodal contrastive method to explore more reliable multimodal
representations under the weak supervision of unimodal predicting.
Specifically, we first capture task-related unimodal representations and the
unimodal predictions from the introduced unimodal predicting task. Then the
unimodal representations are aligned with the more effective one by the
designed multimodal contrastive method under the supervision of the unimodal
predictions. Experimental results with fused features on two image-text
classification benchmarks UPMC-Food-101 and N24News show that our proposed
Unimodality-Supervised MultiModal Contrastive UniS-MMC learning method
outperforms current state-of-the-art multimodal methods. The detailed ablation
study and analysis further demonstrate the advantage of our proposed method.

本文提出了一种新的多模态对比方法，以无单模预测的弱监督下探索更可靠的多模态表示，实验证明该方法优于当前最先进的多模态方法，适用于图像 - 文本分类。