We present an approach to efficiently and effectively adapt a masked image
modeling (MIM) pre-trained vanilla Vision Transformer (ViT) for object
detection, which is based on our two novel observations: (i) A MIM pre-trained
vanilla ViT encoder can work surprisingly well in the challenging object-level
recognition scenario even with randomly sampled partial observations, e.g.,
only 25% $\sim$ 50% of the input embeddings. (ii) In order to construct
multi-scale representations for object detection from single-scale ViT, a
randomly initialized compact convolutional stem supplants the pre-trained large
kernel patchify stem, and its intermediate features can naturally serve as the
higher resolution inputs of a feature pyramid network without further
upsampling or other manipulations. While the pre-trained ViT is only regarded
as the 3$^{rd}$-stage of our detector's backbone instead of the whole feature
extractor. This results in a ConvNet-ViT hybrid feature extractor. The proposed
detector, named MIMDet, enables a MIM pre-trained vanilla ViT to outperform
hierarchical Swin Transformer by 2.5 box AP and 2.6 mask AP on COCO, and
achieves better results compared with the previous best adapted vanilla ViT
detector using a more modest fine-tuning recipe while converging 2.8$\times$
faster. Code and pre-trained models are available at
this https URL

本文提出了 MIMDet 检测器，采用预处理的 ViT 编码器作为检测器基础，通过嵌入卷积中间特征构建多尺度表示，最终结果比采用较为保守微调的 ViT 检测器在 COCO 上优于 2.5 个盒子 AP 和 2.6 个掩码 AP，并且收敛速度更快。