Contrastive language image pretraining (CLIP) is a standard method for
training vision-language models. While CLIP is scalable, promptable, and robust
to distribution shifts on image classification tasks, it lacks object
localization capabilities. This paper studies the following question: Can we
augment CLIP training with task-specific vision models from model zoos to
improve its visual representations? Towards this end, we leverage open-source
task-specific vision models to generate pseudo-labels for an uncurated and
noisy image-text dataset. Subsequently, we train CLIP models on these
pseudo-labels in addition to the contrastive training on image and text pairs.
This simple setup shows substantial improvements of up to 16.3% across
different vision tasks, including segmentation, detection, depth estimation,
and surface normal estimation. Importantly, these enhancements are achieved
without compromising CLIP's existing capabilities, including its proficiency in
promptable zero-shot classification.

通过在 CLIP 训练中结合任务特定的视觉模型，利用伪标签来改进其视觉表示，该简单的设置在不妨碍现有性能的前提下，显著提高了不同视觉任务的效果。