In this paper, we present a simple yet effective contrastive knowledge
distillation approach, which can be formulated as a sample-wise alignment
problem with intra- and inter-sample constraints. Unlike traditional knowledge
distillation methods that concentrate on maximizing feature similarities or
preserving class-wise semantic correlations between teacher and student
features, our method attempts to recover the "dark knowledge" by aligning
sample-wise teacher and student logits. Specifically, our method first
minimizes logit differences within the same sample by considering their
numerical values, thus preserving intra-sample similarities. Next, we bridge
semantic disparities by leveraging dissimilarities across different samples.
Note that constraints on intra-sample similarities and inter-sample
dissimilarities can be efficiently and effectively reformulated into a
contrastive learning framework with newly designed positive and negative pairs.
The positive pair consists of the teacher's and student's logits derived from
an identical sample, while the negative pairs are formed by using logits from
different samples. With this formulation, our method benefits from the
simplicity and efficiency of contrastive learning through the optimization of
InfoNCE, yielding a run-time complexity that is far less than $O(n^2)$, where
$n$ represents the total number of training samples. Furthermore, our method
can eliminate the need for hyperparameter tuning, particularly related to
temperature parameters and large batch sizes. We conduct comprehensive
experiments on three datasets including CIFAR-100, ImageNet-1K, and MS COCO.
Experimental results clearly confirm the effectiveness of the proposed method
on both image classification and object detection tasks. Our source codes will
be publicly available at this https URL

我们提出了一种简单而有效的对比知识蒸馏方法，通过样本间和样本内约束问题的分析，实现了‘黑暗知识’的恢复。我们的方法通过对齐学生和教师的逐样本逻辑回归，最小化了相同样本中的逻辑回归差异，同时通过利用不同样本之间的差异性来弥合语义差异。通过对这些约束进行对比学习框架的优化，我们的方法不仅具备简单、高效的特点，还能有效地减少运行复杂度和超参数调整的需求。通过在 CIFAR-100、ImageNet-1K 和 MS COCO 三个数据集上的实验证明了我们提出的方法在图像分类和物体检测任务上的有效性。

CKD：基于样本级别的对比知识蒸馏

CKD: Contrastive Knowledge Distillation from A Sample-wise Perspective

Image-based object pose estimation sounds amazing because in real
applications the shape of object is oftentimes not available or not easy to
take like photos. Although it is an advantage to some extent, un-explored shape
information in 3D vision learning problem looks like "flaws in jade". In this
paper, we deal with the problem in a reasonable new setting, namely 3D shape is
exploited in the training process, and the testing is still purely image-based.
We enhance the performance of image-based methods for category-agnostic object
pose estimation by exploiting 3D knowledge learned by a multi-modal method.
Specifically, we propose a novel contrastive knowledge distillation framework
that effectively transfers 3D-augmented image representation from a multi-modal
model to an image-based model. We integrate contrastive learning into the
two-stage training procedure of knowledge distillation, which formulates an
advanced solution to combine these two approaches for cross-modal tasks. We
experimentally report state-of-the-art results compared with existing
category-agnostic image-based methods by a large margin (up to +5% improvement
on ObjectNet3D dataset), demonstrating the effectiveness of our method.

该研究论文提出了一种基于图像的物体姿态估计方法，利用多模态方法学习的 3D 知识通过对比知识蒸馏框架有效地转移给单模态模型，使其在没有 3D 信息的情况下提升物体姿态估计准确率，实验证明了该方法的有效性。