Neural network robustness has become a central topic in machine learning in
recent years. Most training algorithms that improve the model's robustness to
adversarial and common corruptions also introduce a large computational
overhead, requiring as many as ten times the number of forward and backward
passes in order to converge. To combat this inefficiency, we propose
BulletTrain $-$ a boundary example mining technique to drastically reduce the
computational cost of robust training. Our key observation is that only a small
fraction of examples are beneficial for improving robustness. BulletTrain
dynamically predicts these important examples and optimizes robust training
algorithms to focus on the important examples. We apply our technique to
several existing robust training algorithms and achieve a 2.1$\times$ speed-up
for TRADES and MART on CIFAR-10 and a 1.7$\times$ speed-up for AugMix on
CIFAR-10-C and CIFAR-100-C without any reduction in clean and robust accuracy.

提出了一种新的基于边界样例挖掘的技术 BulletTrain，通过动态预测重要样例并优化算法来大幅减少机器学习中对强鲁棒性训练所需的计算成本，并成功将其应用于多个已有的算法，包括 TRADES 和 MART 等，并在 CIFAR-10 和 CIFAR-100-C 的情况下取得了 1.7 倍速度提升。

BulletTrain：通过边界示例挖掘加速鲁棒神经网络训练

BulletTrain: Accelerating Robust Neural Network Training via Boundary  Example Mining

Adversarial Training (AT) with Projected Gradient Descent (PGD) is an
effective approach for improving the robustness of the deep neural networks.
However, PGD AT has been shown to suffer from two main limitations: i) high
computational cost, and ii) extreme overfitting during training that leads to
reduction in model generalization. While the effect of factors such as model
capacity and scale of training data on adversarial robustness have been
extensively studied, little attention has been paid to the effect of a very
important parameter in every network optimization on adversarial robustness:
the learning rate. In particular, we hypothesize that effective learning rate
scheduling during adversarial training can significantly reduce the overfitting
issue, to a degree where one does not even need to adversarially train a model
from scratch but can instead simply adversarially fine-tune a pre-trained
model. Motivated by this hypothesis, we propose a simple yet very effective
adversarial fine-tuning approach based on a $\textit{slow start, fast decay}$
learning rate scheduling strategy which not only significantly decreases
computational cost required, but also greatly improves the accuracy and
robustness of a deep neural network. Experimental results show that the
proposed adversarial fine-tuning approach outperforms the state-of-the-art
methods on CIFAR-10, CIFAR-100 and ImageNet datasets in both test accuracy and
the robustness, while reducing the computational cost by 8-10$\times$.
Furthermore, a very important benefit of the proposed adversarial fine-tuning
approach is that it enables the ability to improve the robustness of any
pre-trained deep neural network without needing to train the model from
scratch, which to the best of the authors' knowledge has not been previously
demonstrated in research literature.

本研究提出了一种基于缓慢上升和快速下降型学习率调度策略的对抗微调方法，该方法通过有效的学习率调度策略显著降低了计算成本，同时提高了深度神经网络的准确性和鲁棒性。 实验结果表明，该方法在 CIFAR-10，CIFAR-100 和 ImageNet 数据集上优于先前的最先进方法，同时将计算成本降低了 8-10 倍，并能够改善任何经过预训练的深度神经网络的鲁棒性，而无需从头开始训练模型。