Self-attention based transformer models have been dominating many computer
vision tasks in the past few years. Their superb model qualities heavily depend
on the excessively large labeled image datasets. In order to reduce the
reliance on large labeled datasets, reconstruction based masked autoencoders
are gaining popularity, which learn high quality transferable representations
from unlabeled images. For the same purpose, recent weakly supervised image
pretraining methods explore language supervision from text captions
accompanying the images. In this work, we propose masked image pretraining on
language assisted representation, dubbed as MILAN. Instead of predicting raw
pixels or low level features, our pretraining objective is to reconstruct the
image features with substantial semantic signals that are obtained using
caption supervision. Moreover, to accommodate our reconstruction target, we
propose a more effective prompting decoder architecture and a semantic aware
mask sampling mechanism, which further advance the transfer performance of the
pretrained model. Experimental results demonstrate that MILAN delivers higher
accuracy than the previous works. When the masked autoencoder is pretrained and
finetuned on ImageNet-1K dataset with an input resolution of 224x224, MILAN
achieves a top-1 accuracy of 85.4% on ViT-Base, surpassing previous
state-of-the-arts by 1%. In the downstream semantic segmentation task, MILAN
achieves 52.7 mIoU using ViT-Base on ADE20K dataset, outperforming previous
masked pretraining results by 4 points.

本文提出了一种基于自注意力和掩码自编码器的图像预训练方法 MILAN，通过嵌入语言监督来生成语义信息丰富的图像特征，通过实验证明，该方法在多项计算机视觉任务中优于现有方法。