Self-supervised learning (SSL) with vision transformers (ViTs) has proven
effective for representation learning as demonstrated by the impressive
performance on various downstream tasks. Despite these successes, existing
ViT-based SSL architectures do not fully exploit the ViT backbone, particularly
the patch tokens of the ViT. In this paper, we introduce a novel Semantic Graph
Consistency (SGC) module to regularize ViT-based SSL methods and leverage patch
tokens effectively. We reconceptualize images as graphs, with image patches as
nodes and infuse relational inductive biases by explicit message passing using
Graph Neural Networks into the SSL framework. Our SGC loss acts as a
regularizer, leveraging the underexploited patch tokens of ViTs to construct a
graph and enforcing consistency between graph features across multiple views of
an image. Extensive experiments on various datasets including ImageNet, RESISC
and Food-101 show that our approach significantly improves the quality of
learned representations, resulting in a 5-10\% increase in performance when
limited labeled data is used for linear evaluation. These experiments coupled
with a comprehensive set of ablations demonstrate the promise of our approach
in various settings.

我们介绍了一种新的语义图一致性（SGC）模块，利用视觉转换器（ViTs）的路径标记来规范 ViT 基于自监督学习（SSL）方法，并有效利用路径标记。我们将图像重新理论化为图形，将图像补丁作为节点，并通过显式的图神经网络消息传递将关系感应偏置纳入 SSL 框架。我们的语义图一致性损失作为正则化器，利用 ViTs 的未充分利用的路径标记构建图形，并强制在图像的多个视图之间保持图形特征的一致性。包括 ImageNet、RESISC 和 Food-101 的广泛实验表明我们的方法大大提高了所学特征的质量，当使用有限的标记数据进行线性评估时，性能提升了 5-10%。这些实验以及一整套全面测试表明了我们方法在各种情况下的潜力。