The learning objective of vision-language approach of CLIP does not
effectively account for the noisy many-to-many correspondences found in
web-harvested image captioning datasets, which contributes to its compute and
data inefficiency. To address this challenge, we introduce a novel training
framework based on cross-modal contrastive learning that uses progressive
self-distillation and soft image-text alignments to more efficiently learn
robust representations from noisy data. Our model distills its own knowledge to
dynamically generate soft-alignment targets for a subset of images and captions
in every minibatch, which are then used to update its parameters. Extensive
evaluation across 14 benchmark datasets shows that our method consistently
outperforms its CLIP counterpart in multiple settings, including: (a) zero-shot
classification, (b) linear probe transfer, and (c) image-text retrieval,
without incurring added computational cost. Analysis using an ImageNet-based
robustness test-bed reveals that our method offers better effective robustness
to natural distribution shifts compared to both ImageNet-trained models and
CLIP itself. Lastly, pretraining with datasets spanning two orders of magnitude
in size shows that our improvements over CLIP tend to scale with number of
training examples.

通过交叉模态对比学习以及软图像 - 文本对齐等方法，改进了 CLIP 模型，在处理带有噪声的数据集时能更加高效地学习出具有鲁棒性的表示。经过对 14 个基准数据集的广泛评估，该方法在多种设置下表现均优于 CLIP，并且没有增加计算成本。此外，该方法还在自然分布偏移的鲁棒性测试中表现更好。