Knowledge distillation (KD) improves the performance of a low-complexity
student model with the help of a more powerful teacher. The teacher in KD is a
black-box model, imparting knowledge to the student only through its
predictions. This limits the amount of transferred knowledge. In this work, we
introduce a novel Knowledge Explaining Distillation (KED) framework, which
allows the student to learn not only from the teacher's predictions but also
from the teacher's explanations. We propose a class of superfeature-explaining
teachers that provide explanation over groups of features, along with the
corresponding student model. We also present a method for constructing the
superfeatures. We then extend KED to reduce complexity in convolutional neural
networks, to allow augmentation with hidden-representation distillation
methods, and to work with a limited amount of training data using chimeric
sets. Our experiments over a variety of datasets show that KED students can
substantially outperform KD students of similar complexity.

我们提出了一种名为知识解释蒸馏（KED）的框架，通过引入超特征解释教师，允许学生不仅从教师的预测中学习，还从教师的解释中学习，以及使用卷积神经网络降低复杂性、隐藏表示蒸馏方法扩充和使用嵌套数据集来处理有限的训练数据，实验证明，KED 学生可以显著优于类似复杂度的 KD 学生。