The primary goal of knowledge distillation (KD) is to encapsulate the
information of a model learned from a teacher network into a student network,
with the latter being more compact than the former. Existing work, e.g., using
Kullback-Leibler divergence for distillation, may fail to capture important
structural knowledge in the teacher network and often lacks the ability for
feature generalization, particularly in situations when teacher and student are
built to address different classification tasks. We propose Wasserstein
Contrastive Representation Distillation (WCoRD), which leverages both primal
and dual forms of Wasserstein distance for KD. The dual form is used for global
knowledge transfer, yielding a contrastive learning objective that maximizes
the lower bound of mutual information between the teacher and the student
networks. The primal form is used for local contrastive knowledge transfer
within a mini-batch, effectively matching the distributions of features between
the teacher and the student networks. Experiments demonstrate that the proposed
WCoRD method outperforms state-of-the-art approaches on privileged information
distillation, model compression and cross-modal transfer.

本文提出了一种融合 Wasserstein 距离和对比学习的知识蒸馏方法 WCoRD，可以在压缩模型和跨模态迁移等任务上超越现有的方法。