Self-supervised learning (SSL) has delivered superior performance on a
variety of downstream vision tasks. Two main-stream SSL frameworks have been
proposed, i.e., Instance Discrimination (ID) and Masked Image Modeling (MIM).
ID pulls together representations from different views of the same image, while
avoiding feature collapse. It lacks spatial sensitivity, which requires
modeling the local structure within each image. On the other hand, MIM
reconstructs the original content given a masked image. It instead does not
have good semantic alignment, which requires projecting semantically similar
views into nearby representations. To address this dilemma, we observe that (1)
semantic alignment can be achieved by matching different image views with
strong augmentations; (2) spatial sensitivity can benefit from predicting dense
representations with masked images. Driven by these analysis, we propose
Siamese Image Modeling (SiameseIM), which predicts the dense representations of
an augmented view, based on another masked view from the same image but with
different augmentations. SiameseIM uses a Siamese network with two branches.
The online branch encodes the first view, and predicts the second view's
representation according to the relative positions between these two views. The
target branch produces the target by encoding the second view. SiameseIM can
surpass both ID and MIM on a wide range of downstream tasks, including ImageNet
finetuning and linear probing, COCO and LVIS detection, and ADE20k semantic
segmentation. The improvement is more significant in few-shot, long-tail and
robustness-concerned scenarios. Code shall be released at
this https URL

该研究提出了一种新的自监督学习方法 Siamese Image Modeling，它通过预测同一图像的不同视图的表示来达到语义对齐和空间敏感的目的，这种方法在 ImageNet 微调和线性探测、COCO 和 LVIS 检测以及 ADE20k 语义分割等多项下游任务中都取得了更好的效果。