Masked Autoencoders (MAE) have been prevailing paradigms for large-scale
vision representation pre-training. By reconstructing masked image patches from
a small portion of visible image regions, MAE forces the model to infer
semantic correlation within an image. Recently, some approaches apply
semantic-rich teacher models to extract image features as the reconstruction
target, leading to better performance. However, unlike the low-level features
such as pixel values, we argue the features extracted by powerful teacher
models already encode rich semantic correlation across regions in an intact
image.This raises one question: is reconstruction necessary in Masked Image
Modeling (MIM) with a teacher model? In this paper, we propose an efficient MIM
paradigm named MaskAlign. MaskAlign simply learns the consistency of visible
patch features extracted by the student model and intact image features
extracted by the teacher model. To further advance the performance and tackle
the problem of input inconsistency between the student and teacher model, we
propose a Dynamic Alignment (DA) module to apply learnable alignment. Our
experimental results demonstrate that masked modeling does not lose
effectiveness even without reconstruction on masked regions. Combined with
Dynamic Alignment, MaskAlign can achieve state-of-the-art performance with much
higher efficiency. Code and models will be available at
this https URL

本文提出了一种名为 MaskAlign 的高效 MIM 范例，将可视化小片段功能与老师模型提取的整体图像特征相一致，证明即使在掩蔽区域没有重建的情况下，蒙面建模也不会失去效率。与 Dynamic Alignment 相结合，MaskAlign 能够实现高效的最新性能。