Knowledge distillation has emerged as a highly effective method for bridging
the representation discrepancy between large-scale models and lightweight
models. Prevalent approaches involve leveraging appropriate metrics to minimize
the divergence or distance between the knowledge extracted from the teacher
model and the knowledge learned by the student model. Centered Kernel Alignment
(CKA) is widely used to measure representation similarity and has been applied
in several knowledge distillation methods. However, these methods are complex
and fail to uncover the essence of CKA, thus not answering the question of how
to use CKA to achieve simple and effective distillation properly. This paper
first provides a theoretical perspective to illustrate the effectiveness of
CKA, which decouples CKA to the upper bound of Maximum Mean Discrepancy~(MMD)
and a constant term. Drawing from this, we propose a novel Relation-Centered
Kernel Alignment~(RCKA) framework, which practically establishes a connection
between CKA and MMD. Furthermore, we dynamically customize the application of
CKA based on the characteristics of each task, with less computational source
yet comparable performance than the previous methods. The extensive experiments
on the CIFAR-100, ImageNet-1k, and MS-COCO demonstrate that our method achieves
state-of-the-art performance on almost all teacher-student pairs for image
classification and object detection, validating the effectiveness of our
approaches.

使用 CKA（Centered Kernel Alignment）理论有效地将大规模模型和轻量级模型之间的表达差异解耦为 MMD（Maximum Mean Discrepancy）的上界和一个常数项，提出了一种新颖的 RCKA（Relation-Centered Kernel Alignment）框架，根据任务特征动态定制 CKA 的应用，以较低的计算源消耗获得相比前期方法相当的性能，在 CIFAR-100、ImageNet-1k 和 MS-COCO 上的广泛实验表明，该方法在几乎所有的师生网络对中对于图像分类和目标检测都达到了最先进的性能。