Image-Text pretraining on web-scale image caption dataset has become the default recipe for open vocabulary classification and retrieval models thanks to the success of CLIP and its variants. Several works have also used CLIP features for dense prediction tasks and have shown the emergence of open-set abilities. However, the contrastive objective only focuses on image-text alignment and does not incentivise image feature learning for dense prediction tasks. In this work, we propose the simple addition of local-to-global correspondence learning by self-distillation as an additional objective for contrastive pre-training to propose SILC. We show that distilling local image features from an exponential moving average (EMA) teacher model significantly improves model performance on several computer vision tasks including classification, retrieval, and especially segmentation. We further show that SILC scales better with the same training duration compared to the baselines. Our model SILC sets a new state of the art for zero-shot classification, few shot classification, image and text retrieval, zero-shot segmentation, and open vocabulary segmentation.

基于对 CLIP 模型的改进，本研究提出了 SILC 方法，通过引入本地到全局对应学习来预训练模型，有效提升了计算机视觉领域中的分类、检索和分割等任务的性能，取得了零样本分类、少样本分类、图像与文本检索、无样本分割以及开放词汇分割等方面的最新技术成果。

SILC：用自我蒸馏提升视觉语言预训练