Knowledge Distillation (KD) is a popular technique to transfer knowledge from
a teacher model or ensemble to a student model. Its success is generally
attributed to the privileged information on similarities/consistency between
the class distributions or intermediate feature representations of the teacher
model and the student model. However, directly pushing the student model to
mimic the probabilities/features of the teacher model to a large extent limits
the student model in learning undiscovered knowledge/features. In this paper,
we propose a novel inheritance and exploration knowledge distillation framework
(IE-KD), in which a student model is split into two parts - inheritance and
exploration. The inheritance part is learned with a similarity loss to transfer
the existing learned knowledge from the teacher model to the student model,
while the exploration part is encouraged to learn representations different
from the inherited ones with a dis-similarity loss. Our IE-KD framework is
generic and can be easily combined with existing distillation or mutual
learning methods for training deep neural networks. Extensive experiments
demonstrate that these two parts can jointly push the student model to learn
more diversified and effective representations, and our IE-KD can be a general
technique to improve the student network to achieve SOTA performance.
Furthermore, by applying our IE-KD to the training of two networks, the
performance of both can be improved w.r.t. deep mutual learning. The code and
models of IE-KD will be make publicly available at
this https URL

本文提出了一种新颖的知识蒸馏策略 -- 继承与探索知识蒸馏框架 (IE-KD)，它将学生模型分成两个部分，继承和探索。继承部分通过相似度损失将已有的知识从教师模型转移到学生模型，而探索部分通过不相似度损失鼓励学生模型学习与继承部分不同的特征表示，实验结果表明，IE-KD 能够提高学生网络的表现，达到 SOTA 水平。