Knowledge distillation (KD) aims at improving the performance of a compact
student model by distilling the knowledge from a high-performing teacher model.
In this paper, we present an adaptive KD approach, namely AdaDistill, for deep
face recognition. The proposed AdaDistill embeds the KD concept into the
softmax loss by training the student using a margin penalty softmax loss with
distilled class centers from the teacher. Being aware of the relatively low
capacity of the compact student model, we propose to distill less complex
knowledge at an early stage of training and more complex one at a later stage
of training. This relative adjustment of the distilled knowledge is controlled
by the progression of the learning capability of the student over the training
iterations without the need to tune any hyper-parameters. Extensive experiments
and ablation studies show that AdaDistill can enhance the discriminative
learning capability of the student and demonstrate superiority over various
state-of-the-art competitors on several challenging benchmarks, such as IJB-B,
IJB-C, and ICCV2021-MFR

使用自适应知识蒸馏方法 AdaDistill 在深度人脸识别中，通过利用高性能教师模型的知识来提高紧凑学生模型的性能。AdaDistill 将知识蒸馏概念嵌入到带有蒸馏类中心的边际惩罚 softmax 损失中，通过在训练迭代中控制学生的学习能力进展来相对调整蒸馏的知识，无需调整任何超参数。大量实验证明了 AdaDistill 可以增强学生的判别学习能力，并在多个具有挑战性的基准测试中展示了对各种最先进竞争者的优越性。