Masked image modeling (MIM) as pre-training is shown to be effective for
numerous vision downstream tasks, but how and where MIM works remain unclear.
In this paper, we compare MIM with the long-dominant supervised pre-trained
models from two perspectives, the visualizations and the experiments, to
uncover their key representational differences. From the visualizations, we
find that MIM brings locality inductive bias to all layers of the trained
models, but supervised models tend to focus locally at lower layers but more
globally at higher layers. That may be the reason why MIM helps Vision
Transformers that have a very large receptive field to optimize. Using MIM, the
model can maintain a large diversity on attention heads in all layers. But for
supervised models, the diversity on attention heads almost disappears from the
last three layers and less diversity harms the fine-tuning performance. From
the experiments, we find that MIM models can perform significantly better on
geometric and motion tasks with weak semantics or fine-grained classification
tasks, than their supervised counterparts. Without bells and whistles, a
standard MIM pre-trained SwinV2-L could achieve state-of-the-art performance on
pose estimation (78.9 AP on COCO test-dev and 78.0 AP on CrowdPose), depth
estimation (0.287 RMSE on NYUv2 and 1.966 RMSE on KITTI), and video object
tracking (70.7 SUC on LaSOT). For the semantic understanding datasets where the
categories are sufficiently covered by the supervised pre-training, MIM models
can still achieve highly competitive transfer performance. With a deeper
understanding of MIM, we hope that our work can inspire new and solid research
in this direction.

本研究通过可视化和实验的角度比较了遮蔽图像模型（MIM）和长期优势的监督式预训练模型的关键表现差异，发现 MIM 可以在所有训练模型的层上引入位置归纳偏差并保持所有层的多样性，从而在较弱语义或细粒度分类任务中表现出色。

揭示面具图像建模的黑暗秘密

Revealing the Dark Secrets of Masked Image Modeling

Transformers have shown great potential in various computer vision tasks
owing to their strong capability in modeling long-range dependency using the
self-attention mechanism. Nevertheless, vision transformers treat an image as
1D sequence of visual tokens, lacking an intrinsic inductive bias (IB) in
modeling local visual structures and dealing with scale variance.
Alternatively, they require large-scale training data and longer training
schedules to learn the IB implicitly. In this paper, we propose a novel Vision
Transformer Advanced by Exploring intrinsic IB from convolutions, ie, ViTAE.
Technically, ViTAE has several spatial pyramid reduction modules to downsample
and embed the input image into tokens with rich multi-scale context by using
multiple convolutions with different dilation rates. In this way, it acquires
an intrinsic scale invariance IB and is able to learn robust feature
representation for objects at various scales. Moreover, in each transformer
layer, ViTAE has a convolution block in parallel to the multi-head
self-attention module, whose features are fused and fed into the feed-forward
network. Consequently, it has the intrinsic locality IB and is able to learn
local features and global dependencies collaboratively. Experiments on ImageNet
as well as downstream tasks prove the superiority of ViTAE over the baseline
transformer and concurrent works. Source code and pretrained models will be
available at GitHub.

论文提出了一种新的具有局部偏好和尺度不变性的图像模型 ——ViTAE，其利用多个卷积模块降采样并将输入图像嵌入到多尺度上下文具有丰富的令牌中，从而学习到具有鲁棒性的对象特征表示。