We introduce Corrupted Image Modeling (CIM) for self-supervised visual
pre-training. CIM uses an auxiliary generator with a small trainable BEiT to
corrupt the input image instead of using artificial [MASK] tokens, where some
patches are randomly selected and replaced with plausible alternatives sampled
from the BEiT output distribution. Given this corrupted image, an enhancer
network learns to either recover all the original image pixels, or predict
whether each visual token is replaced by a generator sample or not. The
generator and the enhancer are simultaneously trained and synergistically
updated. After pre-training, the enhancer can be used as a high-capacity visual
encoder for downstream tasks. CIM is a general and flexible visual pre-training
framework that is suitable for various network architectures. For the first
time, CIM demonstrates that both ViT and CNN can learn rich visual
representations using a unified, non-Siamese framework. Experimental results
show that our approach achieves compelling results in vision benchmarks, such
as ImageNet classification and ADE20K semantic segmentation.

本篇论文介绍了 Corrupted Image Modeling (CIM) 用于图像自监督预训练，通过使用额外的生成器和小型可训练 BEiT 对输入图像进行损坏来实现，而不是使用人工 MASK 令牌，并在训练后可以将增强器用作下游任务的高容量视觉编码器。CIM 是一种通用且灵活的视觉预训练框架，适用于各种网络结构，使用非 Siamese 框架首次证明了 ViT 和 CNN 都可以学习到丰富的视觉表示，并在图像分类和语义分割方面取得了令人满意的结果。