Adversarial training is a practical approach for improving the robustness of
deep neural networks against adversarial attacks. Although bringing reliable
robustness, the performance toward clean examples is negatively affected after
adversarial training, which means a trade-off exists between accuracy and
robustness. Recently, some studies have tried to use knowledge distillation
methods in adversarial training, achieving competitive performance in improving
the robustness but the accuracy for clean samples is still limited. In this
paper, to mitigate the accuracy-robustness trade-off, we introduce the
Multi-Teacher Adversarial Robustness Distillation (MTARD) to guide the model's
adversarial training process by applying a strong clean teacher and a strong
robust teacher to handle the clean examples and adversarial examples,
respectively. During the optimization process, to ensure that different
teachers show similar knowledge scales, we design the Entropy-Based Balance
algorithm to adjust the teacher's temperature and keep the teachers'
information entropy consistent. Besides, to ensure that the student has a
relatively consistent learning speed from multiple teachers, we propose the
Normalization Loss Balance algorithm to adjust the learning weights of
different types of knowledge. A series of experiments conducted on public
datasets demonstrate that MTARD outperforms the state-of-the-art adversarial
training and distillation methods against various adversarial attacks.

本文提出了一种多教师对抗鲁棒性蒸馏的方法 (MTARD)，以指导模型的对抗训练过程，并通过熵平衡算法和归一化损失平衡算法来解决准确性和鲁棒性之间的平衡问题。实验证明，MTARD 在公共数据集上优于现有的对抗训练和蒸馏方法。