Benefiting from well-trained deep neural networks (DNNs), model compression
have captured special attention for computing resource limited equipment,
especially edge devices. Knowledge distillation (KD) is one of the widely used
compression techniques for edge deployment, by obtaining a lightweight student
model from a well-trained teacher model released on public platforms. However,
it has been empirically noticed that the backdoor in the teacher model will be
transferred to the student model during the process of KD. Although numerous KD
methods have been proposed, most of them focus on the distillation of a
high-performing student model without robustness consideration. Besides, some
research adopts KD techniques as effective backdoor mitigation tools, but they
fail to perform model compression at the same time. Consequently, it is still
an open problem to well achieve two objectives of robust KD, i.e., student
model's performance and backdoor mitigation. To address these issues, we
propose RobustKD, a robust knowledge distillation that compresses the model
while mitigating backdoor based on feature variance. Specifically, RobustKD
distinguishes the previous works in three key aspects: (1) effectiveness: by
distilling the feature map of the teacher model after detoxification, the main
task performance of the student model is comparable to that of the teacher
model; (2) robustness: by reducing the characteristic variance between the
teacher model and the student model, it mitigates the backdoor of the student
model under backdoored teacher model scenario; (3) generic: RobustKD still has
good performance in the face of multiple data models (e.g., WRN 28-4,
Pyramid-200) and diverse DNNs (e.g., ResNet50, MobileNet).

RobustKD 是基于特征差异的鲁棒知识蒸馏方法，通过压缩模型并减少学生模型和教师模型之间的特征差异，实现了学生模型的性能和后门缓解的双重目标。