There has been significant progress in Masked Image Modeling (MIM). Existing
MIM methods can be broadly categorized into two groups based on the
reconstruction target: pixel-based and tokenizer-based approaches. The former
offers a simpler pipeline and lower computational cost, but it is known to be
biased toward high-frequency details. In this paper, we provide a set of
empirical studies to confirm this limitation of pixel-based MIM and propose a
new method that explicitly utilizes low-level features from shallow layers to
aid pixel reconstruction. By incorporating this design into our base method,
MAE, we reduce the wasted modeling capability of pixel-based MIM, improving its
convergence and achieving non-trivial improvements across various downstream
tasks. To the best of our knowledge, we are the first to systematically
investigate multi-level feature fusion for isotropic architectures like the
standard Vision Transformer (ViT). Notably, when applied to a smaller model
(e.g., ViT-S), our method yields significant performance gains, such as 1.2\%
on fine-tuning, 2.8\% on linear probing, and 2.6\% on semantic segmentation.
Code and models are available at this https URL

本文通过一系列实证研究确认了基于像素的遮罩图像建模存在的限制，并提出一种利用来自浅层的低级特征辅助像素重建的新方法。将该设计纳入基本方法 MAE 中，我们减少了基于像素的遮罩图像建模的建模能力浪费，提高了其收敛性并在多个下游任务中取得了实质性的改进。据我们所知，我们是首次系统研究多级特征融合在类似标准 Vision Transformer（ViT）的各向同性架构中的应用。值得注意的是，当应用于较小的模型（例如 ViT-S）时，我们的方法在微调、线性探测和语义分割等方面取得了显著的性能提升。代码和模型可在此 https 链接获得。