In this paper, we present a comparative analysis of various self-supervised
Vision Transformers (ViTs), focusing on their local representative power.
Inspired by large language models, we examine the abilities of ViTs to perform
various computer vision tasks with little to no fine-tuning. We design an
evaluation framework to analyze the quality of local, i.e. patch-level,
representations in the context of few-shot semantic segmentation, instance
identification, object retrieval, and tracking. We discover that contrastive
learning based methods like DINO produce more universal patch representations
that can be immediately applied for downstream tasks with no parameter tuning,
compared to masked image modeling. The embeddings learned using the latter
approach, e.g. in masked autoencoders, have high variance features that harm
distance-based algorithms, such as k-NN, and do not contain useful information
for most downstream tasks. Furthermore, we demonstrate that removing these
high-variance features enhances k-NN by providing an analysis of the benchmarks
for this work and for Scale-MAE, a recent extension of masked autoencoders.
Finally, we find an object instance retrieval setting where DINOv2, a model
pretrained on two orders of magnitude more data, performs worse than its less
compute-intensive counterpart DINO.

自本文中，我们对各种自监督视觉变换器（ViTs）进行了比较分析，重点研究了它们的局部代表能力。我们设计了一个评估框架，分析了在少样本语义分割、实例识别、目标检索和跟踪等背景下的局部表示质量。我们发现，基于对比学习的方法如 DINO 产生了更通用的局部表示，可以立即应用于无参数调整的下游任务，而掩蔽图像建模的方法中学习的嵌入具有高方差特征，对于大多数下游任务没有有用信息。此外，通过对本工作的基准和 Scale-MAE 的分析，我们证明了移除这些高方差特征对 k-NN 算法的改进。最后，我们发现 DINOv2 在多训练数量级的数据上预训练的模型在物体实例检索方面表现比计算消耗较小的 DINO 模型更差。