In this paper, we present an integral pre-training framework based on masked
image modeling (MIM). We advocate for pre-training the backbone and neck
jointly so that the transfer gap between MIM and downstream recognition tasks
is minimal. We make two technical contributions. First, we unify the
reconstruction and recognition necks by inserting a feature pyramid into the
pre-training stage. Second, we complement mask image modeling (MIM) with masked
feature modeling (MFM) that offers multi-stage supervision to the feature
pyramid. The pre-trained models, termed integrally pre-trained transformer
pyramid networks (iTPNs), serve as powerful foundation models for visual
recognition. In particular, the base/large-level iTPN achieves an 86.2%/87.8%
top-1 accuracy on ImageNet-1K, a 53.2%/55.6% box AP on COCO object detection
with 1x training schedule using Mask-RCNN, and a 54.7%/57.7% mIoU on ADE20K
semantic segmentation using UPerHead -- all these results set new records. Our
work inspires the community to work on unifying upstream pre-training and
downstream fine-tuning tasks. Code and the pre-trained models will be released
at this https URL.

本研究提出了一种基于屏蔽图像建模的完整预训练框架，统一复现与识别阶段，并通过深入的技术贡献，在图片分类、物体检测和语义分割方面实现了新的最佳结果。