In this paper, we present a simple yet effective contrastive knowledge
distillation approach, which can be formulated as a sample-wise alignment
problem with intra- and inter-sample constraints. Unlike traditional knowledge
distillation methods that concentrate on maximizing feature similarities or
preserving class-wise semantic correlations between teacher and student
features, our method attempts to recover the "dark knowledge" by aligning
sample-wise teacher and student logits. Specifically, our method first
minimizes logit differences within the same sample by considering their
numerical values, thus preserving intra-sample similarities. Next, we bridge
semantic disparities by leveraging dissimilarities across different samples.
Note that constraints on intra-sample similarities and inter-sample
dissimilarities can be efficiently and effectively reformulated into a
contrastive learning framework with newly designed positive and negative pairs.
The positive pair consists of the teacher's and student's logits derived from
an identical sample, while the negative pairs are formed by using logits from
different samples. With this formulation, our method benefits from the
simplicity and efficiency of contrastive learning through the optimization of
InfoNCE, yielding a run-time complexity that is far less than $O(n^2)$, where
$n$ represents the total number of training samples. Furthermore, our method
can eliminate the need for hyperparameter tuning, particularly related to
temperature parameters and large batch sizes. We conduct comprehensive
experiments on three datasets including CIFAR-100, ImageNet-1K, and MS COCO.
Experimental results clearly confirm the effectiveness of the proposed method
on both image classification and object detection tasks. Our source codes will
be publicly available at this https URL

我们提出了一种简单而有效的对比知识蒸馏方法，通过样本间和样本内约束问题的分析，实现了‘黑暗知识’的恢复。我们的方法通过对齐学生和教师的逐样本逻辑回归，最小化了相同样本中的逻辑回归差异，同时通过利用不同样本之间的差异性来弥合语义差异。通过对这些约束进行对比学习框架的优化，我们的方法不仅具备简单、高效的特点，还能有效地减少运行复杂度和超参数调整的需求。通过在 CIFAR-100、ImageNet-1K 和 MS COCO 三个数据集上的实验证明了我们提出的方法在图像分类和物体检测任务上的有效性。