Recently, significant progress has been made in masked image modeling to
catch up to masked language modeling. However, unlike words in NLP, the lack of
semantic decomposition of images still makes masked autoencoding (MAE)
different between vision and language. In this paper, we explore a potential
visual analogue of words, i.e., semantic parts, and we integrate semantic
information into the training process of MAE by proposing a Semantic-Guided
Masking strategy. Compared to widely adopted random masking, our masking
strategy can gradually guide the network to learn various information, i.e.,
from intra-part patterns to inter-part relations. In particular, we achieve
this in two steps. 1) Semantic part learning: we design a self-supervised part
learning method to obtain semantic parts by leveraging and refining the
multi-head attention of a ViT-based encoder. 2) Semantic-guided MAE (SemMAE)
training: we design a masking strategy that varies from masking a portion of
patches in each part to masking a portion of (whole) parts in an image.
Extensive experiments on various vision tasks show that SemMAE can learn better
image representation by integrating semantic information. In particular, SemMAE
achieves 84.5% fine-tuning accuracy on ImageNet-1k, which outperforms the
vanilla MAE by 1.4%. In the semantic segmentation and fine-grained recognition
tasks, SemMAE also brings significant improvements and yields the
state-of-the-art performance.

本文提出了一种 Semantic-Guided Masking 策略，通过引入语义部分将语义信息集成到 MAE 训练过程中，以学习更好的图像表示。该训练方法在各种视觉任务中都可以表现出色，特别是在 ImageNet-1k 中实现了 84.5% 的细调准确度，比基本 MAE 高 1.4%。