Knowledge distillation has emerged as a highly effective method for bridging
the representation discrepancy between large-scale models and lightweight
models. Prevalent approaches involve leveraging appropriate metrics to minimize
the divergence or distance between the knowledge extracted from the teacher
model and the knowledge learned by the student model. Centered Kernel Alignment
(CKA) is widely used to measure representation similarity and has been applied
in several knowledge distillation methods. However, these methods are complex
and fail to uncover the essence of CKA, thus not answering the question of how
to use CKA to achieve simple and effective distillation properly. This paper
first provides a theoretical perspective to illustrate the effectiveness of
CKA, which decouples CKA to the upper bound of Maximum Mean Discrepancy~(MMD)
and a constant term. Drawing from this, we propose a novel Relation-Centered
Kernel Alignment~(RCKA) framework, which practically establishes a connection
between CKA and MMD. Furthermore, we dynamically customize the application of
CKA based on the characteristics of each task, with less computational source
yet comparable performance than the previous methods. The extensive experiments
on the CIFAR-100, ImageNet-1k, and MS-COCO demonstrate that our method achieves
state-of-the-art performance on almost all teacher-student pairs for image
classification and object detection, validating the effectiveness of our
approaches.

使用 CKA（Centered Kernel Alignment）理论有效地将大规模模型和轻量级模型之间的表达差异解耦为 MMD（Maximum Mean Discrepancy）的上界和一个常数项，提出了一种新颖的 RCKA（Relation-Centered Kernel Alignment）框架，根据任务特征动态定制 CKA 的应用，以较低的计算源消耗获得相比前期方法相当的性能，在 CIFAR-100、ImageNet-1k 和 MS-COCO 上的广泛实验表明，该方法在几乎所有的师生网络对中对于图像分类和目标检测都达到了最先进的性能。

知识蒸馏中重新思考聚焦核对齐

Rethinking Centered Kernel Alignment in Knowledge Distillation

We propose a principled method for gradient-based regularization of the
critic of GAN-like models trained by adversarially optimizing the kernel of a
Maximum Mean Discrepancy (MMD). We show that controlling the gradient of the
critic is vital to having a sensible loss function, and devise a method to
enforce exact, analytical gradient constraints at no additional cost compared
to existing approximate techniques based on additive regularizers. The new loss
function is provably continuous, and experiments show that it stabilizes and
accelerates training, giving image generation models that outperform
state-of-the art methods on $160 \times 160$ CelebA and $64 \times 64$
unconditional ImageNet.

本文提出了一种基于 MMD 核的 GAN 模型正则化的梯度的原则性方法，证明控制批评者的梯度对于有意义的损失函数至关重要，并设计了一种方法来实施精确、分析梯度约束，使得新的损失函数具有连续性，并证明实验表明它可以稳定和加速训练，为 160 x 160 的 CelebA 数据集和 64 x 64 的无条件 ImageNet 数据集上的图像生成模型提供了优于现有方法的性能。

MMD GANs 的梯度正则化

On gradient regularizers for MMD GANs

We show that the variational representations for f-divergences currently used
in the literature can be tightened. This has implications to a number of
methods recently proposed based on this representation. As an example
application we use our tighter representation to derive a general f-divergence
estimator based on two i.i.d. samples and derive the dual program for this
estimator that performs well empirically. We also point out a connection
between our estimator and MMD.

本文通过收紧文献中已使用的 f-divergences 变分表示方法，提出了更紧的表示方法。作为一个示例应用，我们使用更紧的表示法推导出一种基于两个独立同分布样本的通用的 f-divergence 估计器，并推导出该估计器的对偶程序，在实践中表现良好。我们还指出了该估计器与最大均值差异 (MMD) 之间的联系。