Recent studies pointed out that knowledge distillation (KD) suffers from two
degradation problems, the teacher-student gap and the incompatibility with
strong data augmentations, making it not applicable to training
state-of-the-art models, which are trained with advanced augmentations.
However, we observe that a key factor, i.e., the temperatures in the softmax
functions for generating probabilities of both the teacher and student models,
was mostly overlooked in previous methods. With properly tuned temperatures,
such degradation problems of KD can be much mitigated. However, instead of
relying on a naive grid search, which shows poor transferability, we propose
Meta Knowledge Distillation (MKD) to meta-learn the distillation with learnable
meta temperature parameters. The meta parameters are adaptively adjusted during
training according to the gradients of the learning objective. We validate that
MKD is robust to different dataset scales, different teacher/student
architectures, and different types of data augmentation. With MKD, we achieve
the best performance with popular ViT architectures among compared methods that
use only ImageNet-1K as training data, ranging from tiny to large models. With
ViT-L, we achieve 86.5% with 600 epochs of training, 0.6% better than MAE that
trains for 1,650 epochs.

本文提出了一种元知识蒸馏（MKD）方法，利用可学习的元温度参数进行元学习，通过适应学习目标的梯度自适应地调整元参数以解决知识蒸馏（KD）的退化问题，从而在不同数据集规模、不同网络和不同数据扩增类型上实现了与当下最优秀方法相当的性能表现。