Adversarial training is one effective approach for training robust deep neural networks against adversarial attacks. While being able to bring reliable robustness, adversarial training (AT) methods in general favor high capacity models, i.e., the larger the model the better the robustness. This tends to limit their effectiveness on small models, which are more preferable in scenarios where storage or computing resources are very limited (e.g., mobile devices). In this paper, we leverage the concept of knowledge distillation to improve the robustness of small models by distilling from adversarially trained large models. We first revisit several state-of-the-art AT methods from a distillation perspective and identify one common technique that can lead to improved robustness: the use of robust soft labels -- predictions of a robust model. Following this observation, we propose a novel adversarial robustness distillation method called Robust Soft Label Adversarial Distillation (RSLAD) to train robust small student models. RSLAD fully exploits the robust soft labels produced by a robust (adversarially-trained) large teacher model to guide the student's learning on both natural and adversarial examples in all loss terms. We empirically demonstrate the effectiveness of our RSLAD approach over existing adversarial training and distillation methods in improving the robustness of small models against state-of-the-art attacks including the AutoAttack. We also provide a set of understandings on our RSLAD and the importance of robust soft labels for adversarial robustness distillation.

本研究使用了知识蒸馏的概念来提高小模型的鲁棒性，旨在改善在存储或计算资源非常有限的情况下对小型模型的有效性。通过使用Robust Soft Label Adversarial Distillation（RSLAD）来训练鲁棒的小学生模型，完全利用了由Robust（对抗训练）大教师模型产生的鲁棒软标签来引导学生的学习。该方法在提高小型模型对AutoAttack等最先进攻击的鲁棒性方面表现出优异的效果，并为对抗鲁棒性蒸馏的Robust Soft标签的重要性提供了一组理解。

重新审视对抗性鲁棒性蒸馏：鲁棒的软标签让学生更好