Adversarial training (AT) is a popular method for training robust deep neural
networks (DNNs) against adversarial attacks. Yet, AT suffers from two
shortcomings: (i) the robustness of DNNs trained by AT is highly intertwined
with the size of the DNNs, posing challenges in achieving robustness in smaller
models; and (ii) the adversarial samples employed during the AT process exhibit
poor generalization, leaving DNNs vulnerable to unforeseen attack types. To
address these dual challenges, this paper introduces adversarial training via
adaptive knowledge amalgamation of an ensemble of teachers (AT-AKA). In
particular, we generate a diverse set of adversarial samples as the inputs to
an ensemble of teachers; and then, we adaptively amalgamate the logtis of these
teachers to train a generalized-robust student. Through comprehensive
experiments, we illustrate the superior efficacy of AT-AKA over existing AT
methods and adversarial robustness distillation techniques against cutting-edge
attacks, including AutoAttack.

通过教师集合的知识自适应融合，以解决深度神经网络在对抗样本训练中的两个问题：在小模型中实现稳健性和对未知攻击类型的脆弱性。

通过教师集合的适应性知识融合进行对抗训练

Adversarial Training via Adaptive Knowledge Amalgamation of an Ensemble  of Teachers

Adversarial data augmentation has shown promise for training robust deep
neural networks against unforeseen data shifts or corruptions. However, it is
difficult to define heuristics to generate effective fictitious target
distributions containing "hard" adversarial perturbations that are largely
different from the source distribution. In this paper, we propose a novel and
effective regularization term for adversarial data augmentation. We
theoretically derive it from the information bottleneck principle, which
results in a maximum-entropy formulation. Intuitively, this regularization term
encourages perturbing the underlying source distribution to enlarge predictive
uncertainty of the current model, so that the generated "hard" adversarial
perturbations can improve the model robustness during training. Experimental
results on three standard benchmarks demonstrate that our method consistently
outperforms the existing state of the art by a statistically significant
margin.

本文提出了基于信息瓶颈原理的最大熵正则化方法用于敌对数据增强，通过扩大模型预测不确定性来产生 “难” 的敌对扰动，提升模型鲁棒性，并在三个基准测试中实现了比现有技术显著的优越性能。