Masked image modeling (MIM) has demonstrated impressive results in
self-supervised representation learning by recovering corrupted image patches.
However, most existing studies operate on low-level image pixels, which hinders
the exploitation of high-level semantics for representation models. In this
work, we propose to use a semantic-rich visual tokenizer as the reconstruction
target for masked prediction, providing a systematic way to promote MIM from
pixel-level to semantic-level. Specifically, we propose vector-quantized
knowledge distillation to train the tokenizer, which discretizes a continuous
semantic space to compact codes. We then pretrain vision Transformers by
predicting the original visual tokens for the masked image patches.
Furthermore, we introduce a patch aggregation strategy which associates
discrete image patches to enhance global semantic representation. Experiments
on image classification and semantic segmentation show that BEiT v2 outperforms
all compared MIM methods. On ImageNet-1K (224 size), the base-size BEiT v2
achieves 85.5% top-1 accuracy for fine-tuning and 80.1% top-1 accuracy for
linear probing. The large-size BEiT v2 obtains 87.3% top-1 accuracy for
ImageNet-1K (224 size) fine-tuning, and 56.7% mIoU on ADE20K for semantic
segmentation. The code and pretrained models are available at
this https URL

本文提出使用语义丰富的视觉分词器作为蒙版预测的重构目标，以将蒙版图像建模从像素级别推广到语义级别，包括矢量量化知识蒸馏、预训练 Vision Transformers 和图像聚合策略等，实现了对图像分类和语义分割的超过现有 MIM 方法的表现。