This paper presents a simple yet effective framework MaskCLIP, which
incorporates a newly proposed masked self-distillation into contrastive
language-image pretraining. The core idea of masked self-distillation is to
distill representation from a full image to the representation predicted from a
masked image. Such incorporation enjoys two vital benefits. First, masked
self-distillation targets local patch representation learning, which is
complementary to vision-language contrastive focusing on text-related
representation. Second, masked self-distillation is also consistent with
vision-language contrastive from the perspective of training objective as both
utilize the visual encoder for feature aligning, and thus is able to learn
local semantics getting indirect supervision from the language. We provide
specially designed experiments with a comprehensive analysis to validate the
two benefits. Symmetrically, we also introduce the local semantic supervision
into the text branch, which further improves the pretraining performance. With
extensive experiments, we show that MaskCLIP, when applied to various
challenging downstream tasks, achieves superior results in linear probing,
finetuning, and zero-shot performance with the guidance of the language
encoder. Code will be release at https://github.com/LightDXY/MaskCLIP.

该文介绍了一个名为 MaskCLIP 的简单而又有效的框架，将一种新提出的掩码自蒸馏方法融入了对比语言 - 图像预训练中。该框架主要利用掩码自蒸馏方法学习局部信息传递，这对文本相关表示的对比学习起到补充作用，并且一定程度上能够从自然语言中得到间接监督。同时，在文本分支中也引入了局部语义监督来提高预训练性能，这个框架在不同的任务中取得了很好的效果。