In the field of vision-language contrastive learning, models such as CLIP
capitalize on matched image-caption pairs as positive examples and leverage
within-batch non-matching pairs as negatives. This approach has led to
remarkable outcomes in zero-shot image classification, cross-modal retrieval,
and linear evaluation tasks. We show that the zero-shot classification and
retrieval capabilities of CLIP-like models can be improved significantly
through the introduction of semantically composite examples during pretraining.
Inspired by CutMix in vision categorization, we create semantically composite
image-caption pairs by merging elements from two distinct instances in the
dataset via a novel procedure. Our method fuses the captions and blends 50% of
each image to form a new composite sample. This simple technique (termed CLIP-C
for CLIP Compositions), devoid of any additional computational overhead or
increase in model parameters, significantly improves zero-shot image
classification and cross-modal retrieval. The benefits of CLIP-C are
particularly pronounced in settings with relatively limited pretraining data.

通过引入语义组合样本，我们通过一个简单的技术（称为 CLIP-C），显著改善了零样本图像分类和跨模态检索的能力，而不需要额外的计算开销或模型参数增加。

语义组合提升视觉 - 语言对比学习

Semantic Compositions Enhance Vision-Language Contrastive Learning

Vision-language contrastive learning frameworks like CLIP enable learning
representations from natural language supervision, and provide strong zero-shot
classification capabilities. However, due to the nature of the supervisory
signal in these paradigms, they lack the ability to learn localized features,
leading to degraded performance on dense prediction tasks like segmentation and
detection. On the other hand, self-supervised learning methods have shown the
ability to learn granular representations, complementing the high-level
features in vision-language training. In this work, we present Harmony, a
framework that combines vision-language training with discriminative and
generative self-supervision to learn visual features that can be generalized
across vision downstream tasks. Our framework is specifically designed to work
on web-scraped data by not relying on negative examples and addressing the
one-to-one correspondence issue using soft CLIP targets generated by an EMA
model. We comprehensively evaluate Harmony across various vision downstream
tasks and find that it significantly outperforms the baseline CLIP and the
previously leading joint self and weakly-supervised methods, MaskCLIP and SLIP.
Specifically, when comparing against these methods, Harmony shows superior
performance in fine-tuning and zero-shot classification on ImageNet-1k,
semantic segmentation on ADE20K, and both object detection and instance
segmentation on MS-COCO, when pre-training a ViT-S/16 on CC3M. We also show
that Harmony outperforms other self-supervised learning methods like iBOT and
MAE across all tasks evaluated. On this https URL our
code is publicly available.

本研究中，我们提出了一种名为 Harmony 的框架，它结合了视觉 - 语言训练和辨别性自监督以及生成性自监督，以学习能够在视觉下游任务中被泛化的视觉特征。我们全面评估了 Harmony 在各种视觉下游任务上的性能，并发现它在 ImageNet-1k 上的微调和零样本分类、ADE20K 上的语义分割以及 MS-COCO 上的目标检测和实例分割等任务中明显优于基线 CLIP 和先前领先的联合自监督和弱监督方法 MaskCLIP 和 SLIP。与其他自监督学习方法（如 iBOT 和 MAE）相比，我们还展示了 Harmony 在所有评估任务上的优越性能。