Significant memory and computational requirements of large deep neural
networks restrict their application on edge devices. Knowledge distillation
(KD) is a prominent model compression technique for deep neural networks in
which the knowledge of a trained large teacher model is transferred to a
smaller student model. The success of knowledge distillation is mainly
attributed to its training objective function, which exploits the soft-target
information (also known as "dark knowledge") besides the given regular hard
labels in a training set. However, it is shown in the literature that the
larger the gap between the teacher and the student networks, the more difficult
is their training using knowledge distillation. To address this shortcoming, we
propose an improved knowledge distillation method (called Annealing-KD) by
feeding the rich information provided by the teacher's soft-targets
incrementally and more efficiently. Our Annealing-KD technique is based on a
gradual transition over annealed soft-targets generated by the teacher at
different temperatures in an iterative process, and therefore, the student is
trained to follow the annealed teacher output in a step-by-step manner. This
paper includes theoretical and empirical evidence as well as practical
experiments to support the effectiveness of our Annealing-KD method. We did a
comprehensive set of experiments on different tasks such as image
classification (CIFAR-10 and 100) and NLP language inference with BERT-based
models on the GLUE benchmark and consistently got superior results.

本文介绍了一种名为 Annealing-KD 的改进型知识蒸馏方法，通过渐进式地提供教师模型提供的丰富信息来训练更小的学生模型，从而使得知识蒸馏方法在训练力度较大的学生模型时更有效。作者在不同的任务上进行了实验验证，包括图像分类和自然语言推理等。