Spatially dense self-supervised learning is a rapidly growing problem domain
with promising applications for unsupervised segmentation and pretraining for
dense downstream tasks. Despite the abundance of temporal data in the form of
videos, this information-rich source has been largely overlooked. Our paper
aims to address this gap by proposing a novel approach that incorporates
temporal consistency in dense self-supervised learning. While methods designed
solely for images face difficulties in achieving even the same performance on
videos, our method improves not only the representation quality for videos-but
also images. Our approach, which we call time-tuning, starts from
image-pretrained models and fine-tunes them with a novel self-supervised
temporal-alignment clustering loss on unlabeled videos. This effectively
facilitates the transfer of high-level information from videos to image
representations. Time-tuning improves the state-of-the-art by 8-10% for
unsupervised semantic segmentation on videos and matches it for images. We
believe this method paves the way for further self-supervised scaling by
leveraging the abundant availability of videos. The implementation can be found
here : this https URL

本研究提出了一种名为时间调谐的方法，通过在无标签视频上使用一种新颖的自监督时序对齐聚类损失函数，从而提高视频和图像的表示质量，进而改善了现有最先进方法在无监督语义分割方面的效果。我们相信这种方法为进一步利用丰富的视频资源进行自监督学习的规模化铺平了道路。