The machine learning community has witnessed a drastic change in the training
pipeline, pivoted by those ''foundation models'' with unprecedented scales.
However, the field of adversarial training is lagging behind, predominantly
centered around small model sizes like ResNet-50, and tiny and low-resolution
datasets like CIFAR-10. To bridge this transformation gap, this paper provides
a modern re-examination with adversarial training, investigating its potential
benefits when applied at scale. Additionally, we introduce an efficient and
effective training strategy to enable adversarial training with giant models
and web-scale data at an affordable computing cost. We denote this newly
introduced framework as AdvXL.
Empirical results demonstrate that AdvXL establishes new state-of-the-art
robust accuracy records under AutoAttack on ImageNet-1K. For example, by
training on DataComp-1B dataset, our AdvXL empowers a vanilla ViT-g model to
substantially surpass the previous records of $l_{\infty}$-, $l_{2}$-, and
$l_{1}$-robust accuracy by margins of 11.4%, 14.2% and 12.9%, respectively.
This achievement posits AdvXL as a pioneering approach, charting a new
trajectory for the efficient training of robust visual representations at
significantly larger scales. Our code is available at
this https URL

通过在规模上应用对抗训练，我们引入了一种名为 AdvXL 的高效训练策略，它能够以可承受的计算成本训练巨型模型和网络规模的数据，从而在 ImageNet-1K 上建立了新的鲁棒准确性记录。