Vision-language models trained with contrastive learning on large-scale noisy
data are becoming increasingly popular for zero-shot recognition problems. In
this paper we improve the following three aspects of the contrastive
pre-training pipeline: dataset noise, model initialization and the training
objective. First, we propose a straightforward filtering strategy titled
Complexity, Action, and Text-spotting (CAT) that significantly reduces dataset
size, while achieving improved performance across zero-shot vision-language
tasks. Next, we propose an approach titled Concept Distillation to leverage
strong unimodal representations for contrastive training that does not increase
training complexity while outperforming prior work. Finally, we modify the
traditional contrastive alignment objective, and propose an importance-sampling
approach to up-sample the importance of hard-negatives without adding
additional complexity. On an extensive zero-shot benchmark of 29 tasks, our
Distilled and Hard-negative Training (DiHT) approach improves on 20 tasks
compared to the baseline. Furthermore, for few-shot linear probing, we propose
a novel approach that bridges the gap between zero-shot and few-shot
performance, substantially improving over prior work. Models are available at
this https URL

通过对大规模嘈杂数据进行对比学习训练的视觉 - 语言模型正在变得越来越受欢迎用于零样本识别问题，本文在数据集噪声、模型初始化和训练目标三个方面对对比预训练管道进行了改进。通过 Complexity、Action、and Text-spotting（CAT）筛选策略、概念蒸馏以及重要性采样方法，我们的 DiHT 方法极大地提高了对零样本和几样本线性探测的效果。