Recent self-supervised learning (SSL) methods have shown impressive results
in learning visual representations from unlabeled images. This paper aims to
improve their performance further by utilizing the architectural advantages of
the underlying neural network, as the current state-of-the-art visual pretext
tasks for SSL do not enjoy the benefit, i.e., they are architecture-agnostic.
In particular, we focus on Vision Transformers (ViTs), which have gained much
attention recently as a better architectural choice, often outperforming
convolutional networks for various visual tasks. The unique characteristic of
ViT is that it takes a sequence of disjoint patches from an image and processes
patch-level representations internally. Inspired by this, we design a simple
yet effective visual pretext task, coined SelfPatch, for learning better
patch-level representations. To be specific, we enforce invariance against each
patch and its neighbors, i.e., each patch treats similar neighboring patches as
positive samples. Consequently, training ViTs with SelfPatch learns more
semantically meaningful relations among patches (without using human-annotated
labels), which can be beneficial, in particular, to downstream tasks of a dense
prediction type. Despite its simplicity, we demonstrate that it can
significantly improve the performance of existing SSL methods for various
visual tasks, including object detection and semantic segmentation.
Specifically, SelfPatch significantly improves the recent self-supervised ViT,
DINO, by achieving +1.3 AP on COCO object detection, +1.2 AP on COCO instance
segmentation, and +2.9 mIoU on ADE20K semantic segmentation.

本文设计了一种称为 SelfPatch 的简单而有效的视觉预训练任务，利用 ViT 的特性，在无需人工注释的情况下提高不同类型视觉任务的性能，通过训练神经网络对各种图像的无监督学习来实现。