We introduce two challenging datasets that reliably cause machine learning
model performance to substantially degrade. The datasets are collected with a
simple adversarial filtration technique to create datasets with limited
spurious cues. Our datasets' real-world, unmodified examples transfer to
various unseen models reliably, demonstrating that computer vision models have
shared weaknesses. The first dataset is called ImageNet-A and is like the
ImageNet test set, but it is far more challenging for existing models. We also
curate an adversarial out-of-distribution detection dataset called ImageNet-O,
which is the first out-of-distribution detection dataset created for ImageNet
models. On ImageNet-A a DenseNet-121 obtains around 2% accuracy, an accuracy
drop of approximately 90%, and its out-of-distribution detection performance on
ImageNet-O is near random chance levels. We find that existing data
augmentation techniques hardly boost performance, and using other public
training datasets provides improvements that are limited. However, we find that
improvements to computer vision architectures provide a promising path towards
robust models.

使用对抗性过滤技术构建了两个具有挑战性的数据集，分别是 ImageNet-A 和 ImageNet-O，这两个数据集的出现使得计算机视觉模型的性能大幅下降，而现有的数据增强技术和公共训练数据集的使用带来的改进有限。