Knowledge distillation (KD) is one of the prominent techniques for model
compression. In this method, the knowledge of a large network (teacher) is
distilled into a model (student) with usually significantly fewer parameters.
KD tries to better-match the output of the student model to that of the teacher
model based on the knowledge extracts from the forward pass of the teacher
network. Although conventional KD is effective for matching the two networks
over the given data points, there is no guarantee that these models would match
in other areas for which we do not have enough training samples. In this work,
we address that problem by generating new auxiliary training samples based on
extracting knowledge from the backward pass of the teacher in the areas where
the student diverges greatly from the teacher. We compute the difference
between the teacher and the student and generate new data samples that maximize
the divergence. This is done by perturbing data samples in the direction of the
gradient of the difference between the student and the teacher. Augmenting the
training set by adding this auxiliary improves the performance of KD
significantly and leads to a closer match between the student and the teacher.
Using this approach, when data samples come from a discrete domain, such as
applications of natural language processing (NLP) and language understanding,
is not trivial. However, we show how this technique can be used successfully in
such applications. We evaluated the performance of our method on various tasks
in computer vision and NLP domains and got promising results.

该论文提出了一种新的知识蒸馏方法，通过在教师模型与学生模型差异较大的地方提取知识，在生成新的辅助样本的过程中改善学生模型的性能，从而使教师模型与学生模型更加匹配。这种方法在自然语言处理和计算机视觉等领域得到了良好的实验结果。