Recent masked image modeling (MIM) has received much attention in
self-supervised learning (SSL), which requires the target model to recover the
masked part of the input image. Although MIM-based pre-training methods achieve
new state-of-the-art performance when transferred to many downstream tasks, the
visualizations show that the learned representations are less separable,
especially compared to those based on contrastive learning pre-training. This
inspires us to think whether the linear separability of MIM pre-trained
representation can be further improved, thereby improving the pre-training
performance. Since MIM and contrastive learning tend to utilize different data
augmentations and training strategies, combining these two pretext tasks is not
trivial. In this work, we propose a novel and flexible pre-training framework,
named MimCo, which combines MIM and contrastive learning through two-stage
pre-training. Specifically, MimCo takes a pre-trained contrastive learning
model as the teacher model and is pre-trained with two types of learning
targets: patch-level and image-level reconstruction losses.
Extensive transfer experiments on downstream tasks demonstrate the superior
performance of our MimCo pre-training framework. Taking ViT-S as an example,
when using the pre-trained MoCov3-ViT-S as the teacher model, MimCo only needs
100 epochs of pre-training to achieve 82.53% top-1 finetuning accuracy on
Imagenet-1K, which outperforms the state-of-the-art self-supervised learning
counterparts.

本文提出了一种名为 MimCo 的新型、灵活的预训练框架，通过两阶段的预训练，将 MIM 和对比学习相结合，提高了 MIM 预训练表示的线性可分性，取得了优越的性能。