Existing knowledge distillation methods generally use a teacher-student
approach, where the student network solely learns from a well-trained teacher.
However, this approach overlooks the inherent differences in learning abilities
between the teacher and student networks, thus causing the capacity-gap
problem. To address this limitation, we propose a novel method called SLKD.

本文介绍了一种名为 SLKD 的新方法，用于解决现有知识蒸馏方法中存在的教师 - 学生方法忽视教师和学生网络学习能力差异所导致的容量差距问题。

通过迁移学习能力来改善知识蒸馏

Improving Knowledge Distillation Via Transferring Learning Ability

With ever growing scale of neural models, knowledge distillation (KD)
attracts more attention as a prominent tool for neural model compression.
However, there are counter intuitive observations in the literature showing
some challenging limitations of KD. A case in point is that the best performing
checkpoint of the teacher might not necessarily be the best teacher for
training the student in KD. Therefore, one important question would be how to
find the best checkpoint of the teacher for distillation? Searching through the
checkpoints of the teacher would be a very tedious and computationally
expensive process, which we refer to as the \textit{checkpoint-search problem}.
Moreover, another observation is that larger teachers might not necessarily be
better teachers in KD which is referred to as the \textit{capacity-gap}
problem. To address these challenging problems, in this work, we introduce our
progressive knowledge distillation (Pro-KD) technique which defines a smoother
training path for the student by following the training footprints of the
teacher instead of solely relying on distilling from a single mature
fully-trained teacher. We demonstrate that our technique is quite effective in
mitigating the capacity-gap problem and the checkpoint search problem. We
evaluate our technique using a comprehensive set of experiments on different
tasks such as image classification (CIFAR-10 and CIFAR-100), natural language
understanding tasks of the GLUE benchmark, and question answering (SQuAD 1.1
and 2.0) using BERT-based models and consistently got superior results over
state-of-the-art techniques.

本文介绍了一种名为 Progressive Knowledge Distillation 的技术，通过模仿教师模型的训练轨迹，改善了知识蒸馏中的 “容量差距问题”（capacity-gap problem）和 “检查点搜索问题”（checkpoint-search problem），在不同的任务（如图像分类、自然语言理解等）中，与最先进的技术相比，实验结果始终保持更好的表现。