Vision-specific concepts such as "region" have played a key role in extending
general machine learning frameworks to tasks like object detection. Given the
success of region-based detectors for supervised learning and the progress of
intra-image methods for contrastive learning, we explore the use of regions for
reconstructive pre-training. Starting from Masked Autoencoding (MAE) both as a
baseline and an inspiration, we propose a parallel pre-text task tailored to
address the one-to-many mapping between images and regions. Since such regions
can be generated in an unsupervised way, our approach (R-MAE) inherits the wide
applicability from MAE, while being more "region-aware". We conduct thorough
analyses during the development of R-MAE, and converge on a variant that is
both effective and efficient (1.3% overhead over MAE). Moreover, it shows
consistent quantitative improvements when generalized to various pre-training
data and downstream detection and segmentation benchmarks. Finally, we provide
extensive qualitative visualizations to enhance the understanding of R-MAE's
behaviour and potential. Code will be made available at
this https URL

本研究探讨了基于区域的检测器在无监督学习中的应用，使用掩膜自编码和自生成对抗网络（GAN）等方法进行重构预训练，提高了图像分割的精度和效率。

R-MAE: 区域遇见掩模自编码器

R-MAE: Regions Meet Masked Autoencoders

Recently, self-supervised vision transformers have attracted unprecedented
attention for their impressive representation learning ability. However, the
dominant method, contrastive learning, mainly relies on an instance
discrimination pretext task, which learns a global understanding of the image.
This paper incorporates local feature learning into self-supervised vision
transformers via Reconstructive Pre-training (RePre). Our RePre extends
contrastive frameworks by adding a branch for reconstructing raw image pixels
in parallel with the existing contrastive objective. RePre is equipped with a
lightweight convolution-based decoder that fuses the multi-hierarchy features
from the transformer encoder. The multi-hierarchy features provide rich
supervisions from low to high semantic information, which are crucial for our
RePre. Our RePre brings decent improvements on various contrastive frameworks
with different vision transformer architectures. Transfer performance in
downstream tasks outperforms supervised pre-training and state-of-the-art
(SOTA) self-supervised counterparts.

本文提出了一种基于重构预训练技术 (RePre) 的自监督视觉变压器模型，使用本地特征学习来替代对图像的全局理解，其中重构预训练通过向现有对比性目标添加像素重构分支，使用变压器编码器的多层次特征进行处理，并提供了从低到高语义信息的丰富监督信息；在各种对比学习框架和不同的视觉变换器架构上，该方法带来的转移性能超过了受监督的预训练和最先进的自监督方法。