Having access to multi-modal cues (e.g. vision and audio) empowers some
cognitive tasks to be done faster compared to learning from a single modality.
In this work, we propose to transfer knowledge across heterogeneous modalities,
even though these data modalities may not be semantically correlated. Rather
than directly aligning the representations of different modalities, we compose
audio, image, and video representations across modalities to uncover richer
multi-modal knowledge. Our main idea is to learn a compositional embedding that
closes the cross-modal semantic gap and captures the task-relevant semantics,
which facilitates pulling together representations across modalities by
compositional contrastive learning. We establish a new, comprehensive
multi-modal distillation benchmark on three video datasets: UCF101,
ActivityNet, and VGGSound. Moreover, we demonstrate that our model
significantly outperforms a variety of existing knowledge distillation methods
in transferring audio-visual knowledge to improve video representation
learning. Code is released here: this https URL

本文介绍了一种新的跨模态知识迁移方法，使用组合对比学习来学习复合嵌入，通过学习多模态知识来改善视频表示学习表现。在三个视频数据集上进行的实验表明，该方法显著优于现有的知识蒸馏方法。