Training visual embeddings with labeled data supervision has been the de
facto setup for representation learning in computer vision. Inspired by recent
success of adopting masked image modeling (MIM) in self-supervised
representation learning, we propose a simple yet effective setup that can
easily integrate MIM into existing supervised training paradigms. In our
design, in addition to the original classification task applied to a vision
transformer image encoder, we add a shallow transformer-based decoder on top of
the encoder and introduce an MIM task which tries to reconstruct image tokens
based on masked image inputs. We show with minimal change in architecture and
no overhead in inference that this setup is able to improve the quality of the
learned representations for downstream tasks such as classification, image
retrieval, and semantic segmentation. We conduct a comprehensive study and
evaluation of our setup on public benchmarks. On ImageNet-1k, our ViT-B/14
model achieves 81.72% validation accuracy, 2.01% higher than the baseline
model. On K-Nearest-Neighbor image retrieval evaluation with ImageNet-1k, the
same model outperforms the baseline by 1.32%. We also show that this setup can
be easily scaled to larger models and datasets. Code and checkpoints will be
released.

通过将 MIM 集成到现有的监督训练方法中，我们设计了一种简单而有效的方案，通过在视觉转换图像编码器上添加一个浅层的基于 Transformer 的解码器，并引入一个基于遮蔽图像输入的 MIM 任务，来改善下游任务的学习表示质量，如分类、图像检索和语义分割。