Current self-supervised learning (SSL) methods (e.g., SimCLR, DINO, VICReg,
MOCOv3) target primarily on representations at instance level and do not
generalize well to dense prediction tasks, such as object detection and
segmentation. Towards aligning SSL with dense predictions, this paper
demonstrates for the first time the underlying mean-shift clustering process of
Vision Transformers (ViT), which aligns well with natural image semantics
(e.g., a world of objects and stuffs). By employing transformer for joint
embedding and clustering, we propose a two-level feature clustering SSL method,
coined Feature-Level Self-supervised Learning (FLSL). We present the formal
definition of the FLSL problem and construct the objectives from the mean-shift
and k-means perspectives. We show that FLSL promotes remarkable semantic
cluster representations and learns an embedding scheme amenable to intra-view
and inter-view feature clustering. Experiments show that FLSL yields
significant improvements in dense prediction tasks, achieving 44.9 (+2.8)% AP
and 46.5% AP in object detection, as well as 40.8 (+2.3)% AP and 42.1% AP in
instance segmentation on MS-COCO, using Mask R-CNN with ViT-S/16 and ViT-S/8 as
backbone, respectively. FLSL consistently outperforms existing SSL methods
across additional benchmarks, including UAV object detection on UAVDT, and
video instance segmentation on DAVIS 2017. We conclude by presenting
visualization and various ablation studies to better 20 understand the success
of FLSL.

本文探究了使用 Transformer 进行联合嵌入和聚类的一种方法，即两级特征聚类自监督学习（FLSL），该方法在密集预测任务中取得了显著的改进，包括目标检测和实例分割，实验表明 FLSL 方法在各项测试中始终表现优异。