Knowledge distillation~(KD) has proven to be a highly effective approach for enhancing model performance through a teacher-student training scheme. However, most existing distillation methods are designed under the assumption that the teacher and student models belong to the same model family, particularly the hint-based approaches. By using centered kernel alignment (CKA) to compare the learned features between heterogeneous teacher and student models, we observe significant feature divergence. This divergence illustrates the ineffectiveness of previous hint-based methods in cross-architecture distillation. To tackle the challenge in distilling heterogeneous models, we propose a simple yet effective one-for-all KD framework called OFA-KD, which significantly improves the distillation performance between heterogeneous architectures. Specifically, we project intermediate features into an aligned latent space such as the logits space, where architecture-specific information is discarded. Additionally, we introduce an adaptive target enhancement scheme to prevent the student from being disturbed by irrelevant information. Extensive experiments with various architectures, including CNN, Transformer, and MLP, demonstrate the superiority of our OFA-KD framework in enabling distillation between heterogeneous architectures. Specifically, when equipped with our OFA-KD, the student models achieve notable performance improvements, with a maximum gain of 8.0% on the CIFAR-100 dataset and 0.7% on the ImageNet-1K dataset. PyTorch code and checkpoints can be found at https://github.com/Hao840/OFAKD.

通过使用居中核对齐方法比较异构教师和学生模型之间的特征学习情况，我们观察到显著的特征差异，进而揭示了以往基于暗示的方法在跨体系结构蒸馏中的无效性。为了解决在异构模型蒸馏中的挑战，我们提出了一种简单而有效的一对多蒸馏框架（OFA-KD），通过将中间特征投影到对齐的潜空间（如logits空间）中，丢弃体系结构特定信息，并引入自适应目标增强方案来防止学生受到无关信息的干扰。通过多种体系结构的广泛实验，包括CNN、Transformer和MLP，证明了我们OFA-KD框架在实现异构体系结构蒸馏方面的优势。具体而言，在使用我们的OFA-KD框架后，学生模型在CIFAR-100数据集上最多提高了8.0%，在ImageNet-1K数据集上提高了0.7%。

一对多: 在知识蒸馏中弥合异构架构的差距