We introduce the MNIST-C dataset, a comprehensive suite of 15 corruptions
applied to the MNIST test set, for benchmarking out-of-distribution robustness
in computer vision. Through several experiments and visualizations we
demonstrate that our corruptions significantly degrade performance of
state-of-the-art computer vision models while preserving the semantic content
of the test images. In contrast to the popular notion of adversarial
robustness, our model-agnostic corruptions do not seek worst-case performance
but are instead designed to be broad and diverse, capturing multiple failure
modes of modern models. In fact, we find that several previously published
adversarial defenses significantly degrade robustness as measured by MNIST-C.
We hope that our benchmark serves as a useful tool for future work in designing
systems that are able to learn robust feature representations that capture the
underlying semantics of the input.

我们引入了 MNIST-C 数据集，这是一个综合 15 种损坏应用于 MNIST 测试集的数据集，用于评估计算机视觉中的模型的鲁棒性，我们的相关研究显示，我们的损坏显着降低了现代视觉模型的表现能力，而不影响测试图像的语义内容，和先前的对抗性防御相比，我们的模型无关的损坏不寻求最坏情况下的表现，而是设计成广泛多样的，捕捉现代模型的多个失败模式。