Current Vision and Language Models (VLMs) demonstrate strong performance across various vision-language tasks, yet they struggle with fine-grained understanding. This issue stems from weak image-caption alignment in pretraining datasets and a simplified contrastive objective that fails to distinguish nuanced grounding elements such as relations, actions, and attributes. As a result, the models tend to learn bag-of-words representations. To mitigate these challenges, we introduce an intra-modal contrastive loss and a unique cross-modal rank loss with an adaptive threshold that serves as curriculum learning, utilizing our automatically generated hard negatives to augment the model's capacity. Our strategy, which does not necessitate additional annotations or parameters, can be incorporated into any VLM trained with an image-text contrastive loss. Upon application to CLIP, our method leads to significant improvements on three fine-grained benchmarks, and it also enhances the performance of X-VLM, which is the state-of-art moodel on fine-grained reasoning.

为了解决Vision and Language Models对于细致的理解和fine-grained任务的挑战，本研究提出一种基于intra-modal和cross-modal rank loss的策略，该策略不需要额外的注释或参数，可应用于任何使用图像-文本对比损失训练的VLMs。在应用于CLIP上时，该方法在三个细粒度基准测试上显著提高了性能，并增强了X-VLM在细粒度推理上的表现。

对比视觉语言细粒度理解中的同模态句式和排名跨模态硬负例