Cross-modality distillation arises as an important topic for data modalities
containing limited knowledge such as depth maps and high-quality sketches. Such
techniques are of great importance, especially for memory and
privacy-restricted scenarios where labeled training data is generally
unavailable. To solve the problem, existing label-free methods leverage a few
pairwise unlabeled data to distill the knowledge by aligning features or
statistics between the source and target modalities. For instance, one
typically aims to minimize the L2 distance or contrastive loss between the
learned features of pairs of samples in the source (e.g. image) and the target
(e.g. sketch) modalities. However, most algorithms in this domain only focus on
the experimental results but lack theoretical insight. To bridge the gap
between the theory and practical method of cross-modality distillation, we
first formulate a general framework of cross-modality contrastive distillation
(CMCD), built upon contrastive learning that leverages both positive and
negative correspondence, towards a better distillation of generalizable
features. Furthermore, we establish a thorough convergence analysis that
reveals that the distance between source and target modalities significantly
impacts the test error on downstream tasks within the target modality which is
also validated by the empirical results. Extensive experimental results show
that our algorithm outperforms existing algorithms consistently by a margin of
2-3\% across diverse modalities and tasks, covering modalities of image,
sketch, depth map, and audio and tasks of recognition and segmentation.

跨模态蒸馏是一个重要主题，用于包含有限知识的数据模态，如深度图和高质量素描。我们提出了一个基于对比学习的跨模态对比蒸馏（CMCD）的通用框架，该框架利用正负对应关系更好地蒸馏可泛化的特征，并在实验结果和收敛分析中验证了源模态和目标模态之间的距离对目标模态下游任务的测试误差有着显著影响。