This paper revisits the standard pretrain-then-finetune paradigm used in
computer vision for visual recognition tasks. Typically, state-of-the-art
foundation models are pretrained using large scale (weakly) supervised datasets
with billions of images. We introduce an additional pre-pretraining stage that
is simple and uses the self-supervised MAE technique to initialize the model.
While MAE has only been shown to scale with the size of models, we find that it
scales with the size of the training dataset as well. Thus, our MAE-based
pre-pretraining scales with both model and data size making it applicable for
training foundation models. Pre-pretraining consistently improves both the
model convergence and the downstream transfer performance across a range of
model scales (millions to billions of parameters), and dataset sizes (millions
to billions of images). We measure the effectiveness of pre-pretraining on 10
different visual recognition tasks spanning image classification, video
recognition, object detection, low-shot classification and zero-shot
recognition. Our largest model achieves new state-of-the-art results on
iNaturalist-18 (91.3%), 1-shot ImageNet-1k (62.1%), and zero-shot transfer on
Food-101 (96.0%). Our study reveals that model initialization plays a
significant role, even for web-scale pretraining with billions of images.

该论文重新审视了在视觉识别任务中使用的标准预训练 - 微调范式，通过使用自监督 MAE 技术实现额外的预先训练阶段来初始化模型。该方法不仅可以扩展模型大小，还可以扩展训练数据集的规模，从而提高了基础模型的训练效率以及各种视觉识别任务的性能并取得了新的最优结果。