Vision-language models (VLMs) mainly rely on contrastive training to learn
general-purpose representations of images and captions. We focus on the
situation when one image is associated with several captions, each caption
containing both information shared among all captions and unique information
per caption about the scene depicted in the image. In such cases, it is unclear
whether contrastive losses are sufficient for learning task-optimal
representations that contain all the information provided by the captions or
whether the contrastive learning setup encourages the learning of a simple
shortcut that minimizes contrastive loss. We introduce synthetic shortcuts for
vision-language: a training and evaluation framework where we inject synthetic
shortcuts into image-text data. We show that contrastive VLMs trained from
scratch or fine-tuned with data containing these synthetic shortcuts mainly
learn features that represent the shortcut. Hence, contrastive losses are not
sufficient to learn task-optimal representations, i.e., representations that
contain all task-relevant information shared between the image and associated
captions. We examine two methods to reduce shortcut learning in our training
and evaluation framework: (i) latent target decoding and (ii) implicit feature
modification. We show empirically that both methods improve performance on the
evaluation task, but only partly reduce shortcut learning when training and
evaluating with our shortcut learning framework. Hence, we show the difficulty
and challenge of our shortcut learning framework for contrastive
vision-language representation learning.

基于对比训练的视觉语言模型主要依赖于学习图像和标题的通用表示。本文提出了一种新的训练和评估框架，注入了合成的快捷方式来减少学习中的难度和挑战，以改善对比视觉语言表示学习的性能。