Knowledge distillation(KD) aims to improve the performance of a student
network by mimicing the knowledge from a powerful teacher network. Existing
methods focus on studying what knowledge should be transferred and treat all
samples equally during training. This paper introduces the adaptive sample
weighting to KD. We discover that previous effective hard mining methods are
not appropriate for distillation. Furthermore, we propose Prime-Aware Adaptive
Distillation (PAD) by the incorporation of uncertainty learning. PAD perceives
the prime samples in distillation and then emphasizes their effect adaptively.
PAD is fundamentally different from and would refine existing methods with the
innovative view of unequal training. For this reason, PAD is versatile and has
been applied in various tasks including classification, metric learning, and
object detection. With ten teacher-student combinations on six datasets, PAD
promotes the performance of existing distillation methods and outperforms
recent state-of-the-art methods.

本研究提出了一种基于适应样本加权和不确定性学习的智能知识蒸馏方法 PAD，用于改善学生网络的表现。通过 10 个师生组合在 6 个数据集上的测试，PAD 显着提高了现有蒸馏方法的性能，并优于最新的最先进的方法。