Masked image modeling (MIM) as pre-training is shown to be effective for
numerous vision downstream tasks, but how and where MIM works remain unclear.
In this paper, we compare MIM with the long-dominant supervised pre-trained
models from two perspectives, the visualizations and the experiments, to
uncover their key representational differences. From the visualizations, we
find that MIM brings locality inductive bias to all layers of the trained
models, but supervised models tend to focus locally at lower layers but more
globally at higher layers. That may be the reason why MIM helps Vision
Transformers that have a very large receptive field to optimize. Using MIM, the
model can maintain a large diversity on attention heads in all layers. But for
supervised models, the diversity on attention heads almost disappears from the
last three layers and less diversity harms the fine-tuning performance. From
the experiments, we find that MIM models can perform significantly better on
geometric and motion tasks with weak semantics or fine-grained classification
tasks, than their supervised counterparts. Without bells and whistles, a
standard MIM pre-trained SwinV2-L could achieve state-of-the-art performance on
pose estimation (78.9 AP on COCO test-dev and 78.0 AP on CrowdPose), depth
estimation (0.287 RMSE on NYUv2 and 1.966 RMSE on KITTI), and video object
tracking (70.7 SUC on LaSOT). For the semantic understanding datasets where the
categories are sufficiently covered by the supervised pre-training, MIM models
can still achieve highly competitive transfer performance. With a deeper
understanding of MIM, we hope that our work can inspire new and solid research
in this direction.

本研究通过可视化和实验的角度比较了遮蔽图像模型（MIM）和长期优势的监督式预训练模型的关键表现差异，发现 MIM 可以在所有训练模型的层上引入位置归纳偏差并保持所有层的多样性，从而在较弱语义或细粒度分类任务中表现出色。