In this paper, we propose a new self-supervised method, which is called
Denoising Masked AutoEncoders (DMAE), for learning certified robust classifiers
of images. In DMAE, we corrupt each image by adding Gaussian noises to each
pixel value and randomly masking several patches. A Transformer-based
encoder-decoder model is then trained to reconstruct the original image from
the corrupted one. In this learning paradigm, the encoder will learn to capture
relevant semantics for the downstream tasks, which is also robust to Gaussian
additive noises. We show that the pre-trained encoder can naturally be used as
the base classifier in Gaussian smoothed models, where we can analytically
compute the certified radius for any data point. Although the proposed method
is simple, it yields significant performance improvement in downstream
classification tasks. We show that the DMAE ViT-Base model, which just uses
1/10 parameters of the model developed in recent work arXiv:2206.10550,
achieves competitive or better certified accuracy in various settings. The DMAE
ViT-Large model significantly surpasses all previous results, establishing a
new state-of-the-art on ImageNet dataset. We further demonstrate that the
pre-trained model has good transferability to the CIFAR-10 dataset, suggesting
its wide adaptability. Models and code are available at
this https URL

提出了一种自监督的新方法，叫做 Denoising Masked AutoEncoders，可以通过加入高斯噪声和随机掩蔽来训练生成原始图像的编码器 - 解码器模型。使用该模型作为基分类器，能够显著提高下游分类任务的性能，并且在 ImageNet 数据集上建立了新的最先进结果。