Among the ever-evolving development of vision-language models, contrastive
language-image pretraining (CLIP) has set new benchmarks in many downstream
tasks such as zero-shot classifications by leveraging self-supervised
contrastive learning on large amounts of text-image pairs. However, its
dependency on rigid one-to-one mappings overlooks the complex and often
multifaceted relationships between and within texts and images. To this end, we
introduce RankCLIP, a novel pretraining method that extends beyond the rigid
one-to-one matching framework of CLIP and its variants. By leveraging both
in-modal and cross-modal ranking consistency, RankCLIP improves the alignment
process, enabling it to capture the nuanced many-to-many relationships between
and within each modality. Through comprehensive experiments, we demonstrate the
enhanced capability of RankCLIP to effectively improve performance across
various downstream tasks, notably achieving significant gains in zero-shot
classifications over state-of-the-art methods, underscoring the potential of
RankCLIP in further advancing vision-language pretraining.

通过在大量文本图像对上进行自我监督的对比学习，RankCLIP 在扩展 CLIP 的刚性一对一匹配框架的同时，利用模态内和跨模态的排序一致性来提高对齐过程，捕捉每种模态之间和内部的细致的多对多关系，有效提升各种下游任务的性能，尤其在零样本分类方面，显著超越现有方法，突显了 RankCLIP 在进一步推进视觉语言预训练方面的潜力。