In knowledge distillation, the knowledge from the teacher model is often too
complex for the student model to thoroughly process. However, good teachers in
real life always simplify complex material before teaching it to students.
Inspired by this fact, we propose student-friendly knowledge distillation (SKD)
to simplify teacher output into new knowledge representations, which makes the
learning of the student model easier and more effective. SKD contains a
softening processing and a learning simplifier. First, the softening processing
uses the temperature hyperparameter to soften the output logits of the teacher
model, which simplifies the output to some extent and makes it easier for the
learning simplifier to process. The learning simplifier utilizes the attention
mechanism to further simplify the knowledge of the teacher model and is jointly
trained with the student model using the distillation loss, which means that
the process of simplification is correlated with the training objective of the
student model and ensures that the simplified new teacher knowledge
representation is more suitable for the specific student model. Furthermore,
since SKD does not change the form of the distillation loss, it can be easily
combined with other distillation methods that are based on the logits or
features of intermediate layers to enhance its effectiveness. Therefore, SKD
has wide applicability. The experimental results on the CIFAR-100 and ImageNet
datasets show that our method achieves state-of-the-art performance while
maintaining high training efficiency.

本文提出了一种基于教师模型稍加简化后的知识表示的学生友好型知识蒸馏方法（SKD），其包含软化处理和学习简化器，通过联合训练确保知识简化过程与学生模型的训练目标相关，提高了训练效率和准确性。实验结果表明，该方法在 CIFAR-100 和 ImageNet 数据集上取得了最优性能。