Knowledge distillation (KD) exploits a large well-trained model (i.e.,
teacher) to train a small student model on the same dataset for the same task.
Treating teacher features as knowledge, prevailing methods of knowledge
distillation train student by aligning its features with the teacher's, e.g.,
by minimizing the KL-divergence between their logits or L2 distance between
their intermediate features. While it is natural to believe that better
alignment of student features to the teacher better distills teacher knowledge,
simply forcing this alignment does not directly contribute to the student's
performance, e.g., classification accuracy. In this work, we propose to align
student features with class-mean of teacher features, where class-mean
naturally serves as a strong classifier. To this end, we explore baseline
techniques such as adopting the cosine distance based loss to encourage the
similarity between student features and their corresponding class-means of the
teacher. Moreover, we train the student to produce large-norm features,
inspired by other lines of work (e.g., model pruning and domain adaptation),
which find the large-norm features to be more significant. Finally, we propose
a rather simple loss term (dubbed ND loss) to simultaneously (1) encourage
student to produce large-\emph{norm} features, and (2) align the
\emph{direction} of student features and teacher class-means. Experiments on
standard benchmarks demonstrate that our explored techniques help existing KD
methods achieve better performance, i.e., higher classification accuracy on
ImageNet and CIFAR100 datasets, and higher detection precision on COCO dataset.
Importantly, our proposed ND loss helps the most, leading to the
state-of-the-art performance on these benchmarks. The source code is available
at https://github.com/WangYZ1608/Knowledge-Distillation-via-ND.

提出了一种基于大范数特征和类均值对齐的知识蒸馏方法 NDKD，该方法在 ImageNet 和 CIFAR100 数据集上取得了最好的分类精度。