Contrastive Language-Image Pretraining (CLIP) models maximize the mutual information between text and visual modalities to learn representations. This makes the nature of the training data a significant factor in the efficacy of CLIP for downstream tasks. However, the lack of compositional diversity in contemporary image-text datasets limits the compositional reasoning ability of CLIP. We show that generating ``hard'' negative captions via in-context learning and synthesizing corresponding negative images with text-to-image generators offers a solution. We introduce a novel contrastive pre-training strategy that leverages these hard negative captions and images in an alternating fashion to train CLIP. We demonstrate that our method, named TripletCLIP, when applied to existing datasets such as CC3M and CC12M, enhances the compositional capabilities of CLIP, resulting in an absolute improvement of over 9% on the SugarCrepe benchmark on an equal computational budget, as well as improvements in zero-shot image classification and image retrieval. Our code, models, and data are available at: https://tripletclip.github.io

本研究针对当前图像-文本数据集中缺乏组合多样性的问题，提出了一种新的对比预训练策略TripletCLIP。该方法通过在上下文学习中生成“硬”负向描述以及合成相应的负向图像，显著提高了CLIP的组合推理能力，在SugarCrepe基准测试中实现了超过9%的绝对提升，并改善了零样本图像分类和图像检索性能。

TripletCLIP：通过合成视觉-语言负例改善CLIP的组合推理能力