Knowledge distillation aims to compress a powerful yet cumbersome teacher
model into a lightweight student model without much sacrifice of performance.
For this purpose, various approaches have been proposed over the past few
years, generally with elaborately designed knowledge representations, which in
turn increase the difficulty of model development and interpretation. In
contrast, we empirically show that a simple knowledge distillation technique is
enough to significantly narrow down the teacher-student performance gap. We
directly reuse the discriminative classifier from the pre-trained teacher model
for student inference and train a student encoder through feature alignment
with a single $\ell_2$ loss. In this way, the student model is able to achieve
exactly the same performance as the teacher model provided that their extracted
features are perfectly aligned. An additional projector is developed to help
the student encoder match with the teacher classifier, which renders our
technique applicable to various teacher and student architectures. Extensive
experiments demonstrate that our technique achieves state-of-the-art results at
the modest cost of compression ratio due to the added projector.

使用简单的知识蒸馏技术可以显著缩小教师模型与学生模型之间的性能差距，通过使用预先训练的教师模型的判别分类器进行学生推断，并通过特征对齐训练学生编码器来实现与教师相同的性能。添加新的投影仪使学生编码器与教师分类器匹配，从而将这种技术应用于各种教师和学生架构下达到良好的压缩率与状态的最佳结果。