Recently, masked image modeling (MIM) has become a promising direction for
visual pre-training. In the context of vision transformers, MIM learns
effective visual representation by aligning the token-level features with a
pre-defined space (e.g., BEIT used a d-VAE trained on a large image corpus as
the tokenizer). In this paper, we go one step further by introducing guidance
from other modalities and validating that such additional knowledge leads to
impressive gains for visual pre-training. The proposed approach is named
Multimodality-guided Visual Pre-training (MVP), in which we replace the
tokenizer with the vision branch of CLIP, a vision-language model pre-trained
on 400 million image-text pairs. We demonstrate the effectiveness of MVP by
performing standard experiments, i.e., pre-training the ViT models on ImageNet
and fine-tuning them on a series of downstream visual recognition tasks. In
particular, pre-training ViT-Base/16 for 300 epochs, MVP reports a 52.4% mIoU
on ADE20K, surpassing BEIT (the baseline and previous state-of-the-art) with an
impressive margin of 6.8%.

本文介绍了一种名为 Multimodality-guided Visual Pre-training (MVP) 的方法，该方法利用来自其他几个模态的指导信息对图像进行预训练，取代了 Vision Transformer 中的 tokenizer，并在一系列下游视觉识别任务中取得了显著优越的效果。