Adversarial training significantly improves adversarial robustness, but
superior performance is primarily attained with large models. This substantial
performance gap for smaller models has spurred active research into adversarial
distillation (AD) to mitigate the difference. Existing AD methods leverage the
teacher's logits as a guide. In contrast to these approaches, we aim to
transfer another piece of knowledge from the teacher, the input gradient. In
this paper, we propose a distillation module termed Indirect Gradient
Distillation Module (IGDM) that indirectly matches the student's input gradient
with that of the teacher. We hypothesize that students can better acquire the
teacher's knowledge by matching the input gradient. Leveraging the observation
that adversarial training renders the model locally linear on the input space,
we employ Taylor approximation to effectively align gradients without directly
calculating them. Experimental results show that IGDM seamlessly integrates
with existing AD methods, significantly enhancing the performance of all AD
methods. Particularly, utilizing IGDM on the CIFAR-100 dataset improves the
AutoAttack accuracy from 28.06% to 30.32% with the ResNet-18 model and from
26.18% to 29.52% with the MobileNetV2 model when integrated into the SOTA
method without additional data augmentation. The code will be made available.

使用间接梯度蒸馏模块（IGDM）通过匹配学生的输入梯度和教师的输入梯度来改善对抗性模型的性能，实验证明 IGDM 与现有的蒸馏方法无缝集成，显著提高了所有蒸馏方法的性能。