Knowledge distillation is effective for producing small high-performance neural networks for classification, but these small networks are vulnerable to adversarial attacks. We first study how robustness transfers from robust teacher to student network during knowledge distillation. We find that a large amount of robustness may be inherited by the student even when distilled on only clean images. Second, we introduce Adversarially Robust Distillation (ARD) for distilling robustness onto small student networks. ARD is an analogue of adversarial training but for distillation. In addition to producing small models with high test accuracy like conventional distillation, ARD also passes the superior robustness of large networks onto the student. In our experiments, we find that ARD student models decisively outperform adversarially trained networks of identical architecture on robust accuracy. Finally, we adapt recent fast adversarial training methods to ARD for accelerated robust distillation.

本文研究知识蒸馏过程中，如何将教师神经网络的鲁棒性传递给学生神经网络，并提出一种称为 Adversarially Robust Distillation (ARD) 的方法。实验证明，采用 ARD 的学生模型在鲁棒性上的表现明显优于采用相同结构的敌对训练网络，并在标准鲁棒性基准测试中超越了当前最先进的方法。

对抗鲁棒蒸馏