We present a simple self-supervised method to enhance the performance of ViT
features for dense downstream tasks. Our Lightweight Feature Transform (LiFT)
is a straightforward and compact postprocessing network that can be applied to
enhance the features of any pre-trained ViT backbone. LiFT is fast and easy to
train with a self-supervised objective, and it boosts the density of ViT
features for minimal extra inference cost. Furthermore, we demonstrate that
LiFT can be applied with approaches that use additional task-specific
downstream modules, as we integrate LiFT with ViTDet for COCO detection and
segmentation. Despite the simplicity of LiFT, we find that it is not simply
learning a more complex version of bilinear interpolation. Instead, our LiFT
training protocol leads to several desirable emergent properties that benefit
ViT features in dense downstream tasks. This includes greater scale invariance
for features, and better object boundary maps. By simply training LiFT for a
few epochs, we show improved performance on keypoint correspondence, detection,
segmentation, and object discovery tasks. Overall, LiFT provides an easy way to
unlock the benefits of denser feature arrays for a fraction of the
computational cost. For more details, refer to our project page at
this https URL

我们提出了一种简单的自监督方法来提高 ViT 特征在密集下游任务中的性能，通过应用 LiFT 后处理网络，可以提升任何经过预训练的 ViT 主干的特征。LiFT 在自监督目标下训练快速简便，并且在最小额外推理成本下增加了 ViT 特征的密度。此外，我们演示了将 LiFT 与 ViTDet 用于 COCO 检测和分割的方法。尽管 LiFT 方法简单，但我们发现它并不仅仅是学习了一种更复杂的双线性插值版本。相反，我们的 LiFT 训练方案导致了几个有益于密集下游任务中 ViT 特征的理想性质，包括更大的尺度不变性和更好的对象边界映射。通过简单地训练 LiFT 几个 epochs，我们展示了在关键点对应、检测、分割和物体发现任务上改进的性能。总体而言，LiFT 提供了一种以较小的计算代价实现更密集特征阵列优势的简单方法。了解更多细节，请参考我们的项目页面。

LiFT: 一个令人惊讶的简单轻量级特征转换用于稠密 ViT 描述符

LiFT: A Surprisingly Simple Lightweight Feature Transform for Dense ViT  Descriptors

Spatially dense self-supervised learning is a rapidly growing problem domain
with promising applications for unsupervised segmentation and pretraining for
dense downstream tasks. Despite the abundance of temporal data in the form of
videos, this information-rich source has been largely overlooked. Our paper
aims to address this gap by proposing a novel approach that incorporates
temporal consistency in dense self-supervised learning. While methods designed
solely for images face difficulties in achieving even the same performance on
videos, our method improves not only the representation quality for videos-but
also images. Our approach, which we call time-tuning, starts from
image-pretrained models and fine-tunes them with a novel self-supervised
temporal-alignment clustering loss on unlabeled videos. This effectively
facilitates the transfer of high-level information from videos to image
representations. Time-tuning improves the state-of-the-art by 8-10% for
unsupervised semantic segmentation on videos and matches it for images. We
believe this method paves the way for further self-supervised scaling by
leveraging the abundant availability of videos. The implementation can be found
here : this https URL

本研究提出了一种名为时间调谐的方法，通过在无标签视频上使用一种新颖的自监督时序对齐聚类损失函数，从而提高视频和图像的表示质量，进而改善了现有最先进方法在无监督语义分割方面的效果。我们相信这种方法为进一步利用丰富的视频资源进行自监督学习的规模化铺平了道路。