We present Knowledge Distillation with Meta Learning (MetaDistil), a simple
yet effective alternative to traditional knowledge distillation (KD) methods
where the teacher model is fixed during training. We show the teacher network
can learn to better transfer knowledge to the student network (i.e., learning
to teach) with the feedback from the performance of the distilled student
network in a meta learning framework. Moreover, we introduce a pilot update
mechanism to improve the alignment between the inner-learner and meta-learner
in meta learning algorithms that focus on an improved inner-learner.
Experiments on various benchmarks show that MetaDistil can yield significant
improvements compared with traditional KD algorithms and is less sensitive to
the choice of different student capacity and hyperparameters, facilitating the
use of KD on different tasks and models.

本研究提出了用元学习结合知识蒸馏的方法（MetaDistil），可以在固定教师网络期间更好地向学生网络转移知识。在多个实验基准上，MetaDistil 可以比传统的知识蒸馏算法产生更显著的改进，并且对不同的学生容量和超参数的选择不太敏感，有助于在不同的任务和模型上使用知识蒸馏。