Knowledge Distillation (KD) is a powerful technique for transferring
knowledge between neural network models, where a pre-trained teacher model is
used to facilitate the training of the target student model. However, the
availability of a suitable teacher model is not always guaranteed. To address
this challenge, Self-Knowledge Distillation (SKD) attempts to construct a
teacher model from itself. Existing SKD methods add Auxiliary Classifiers (AC)
to intermediate layers of the model or use the history models and models with
different input data within the same class. However, these methods are
computationally expensive and only capture time-wise and class-wise features of
data. In this paper, we propose a lightweight SKD framework that utilizes
multi-source information to construct a more informative teacher. Specifically,
we introduce a Distillation with Reverse Guidance (DRG) method that considers
different levels of information extracted by the model, including edge, shape,
and detail of the input data, to construct a more informative teacher.
Additionally, we design a Distillation with Shape-wise Regularization (DSR)
method that ensures a consistent shape of ranked model output for all data. We
validate the performance of the proposed DRG, DSR, and their combination
through comprehensive experiments on various datasets and models. Our results
demonstrate the superiority of the proposed methods over baselines (up to
2.87%) and state-of-the-art SKD methods (up to 1.15%), while being
computationally efficient and robust. The code is available at
this https URL

提出了一种利用多源信息进行自知识蒸馏的轻量级框架，其中包括反向引导方法和形状正则化方法，其在各种数据集和模型上的综合实验中表现优于基线和最先进的自知识蒸馏方法，同时具有计算效率和鲁棒性。