Aggressive data augmentation is a key component of the strong generalization capabilities of Vision Transformer (ViT). One such data augmentation technique is adversarial training; however, many prior works have shown that this often results in poor clean accuracy. In this work, we present Pyramid Adversarial Training, a simple and effective technique to improve ViT's overall performance. We pair it with a "matched" Dropout and stochastic depth regularization, which adopts the same Dropout and stochastic depth configuration for the clean and adversarial samples. Similar to the improvements on CNNs by AdvProp (not directly applicable to ViT), our Pyramid Adversarial Training breaks the trade-off between in-distribution accuracy and out-of-distribution robustness for ViT and related architectures. It leads to $1.82\%$ absolute improvement on ImageNet clean accuracy for the ViT-B model when trained only on ImageNet-1K data, while simultaneously boosting performance on $7$ ImageNet robustness metrics, by absolute numbers ranging from $1.76\%$ to $11.45\%$. We set a new state-of-the-art for ImageNet-C (41.4 mCE), ImageNet-R ($53.92\%$), and ImageNet-Sketch ($41.04\%$) without extra data, using only the ViT-B/16 backbone and our Pyramid Adversarial Training. Our code will be publicly available upon acceptance.

本研究旨在提高视觉转换器的总体性能，通过实现金字塔对抗训练（PyramidAT），同时采用适配的Dropout和随机深度正则化技术，使在分布和越界数据分类准确性之间的权衡断裂。实验结果显示，当只使用ImageNet-1K数据训练ViT-B模型时，ImageNet图像的清晰度准确性绝对值提高了1.82％，同时提高了7个ImageNet鲁棒性指标，其绝对值范围从1.76％到15.68％。

金字塔对抗训练提高 ViT 模型性能