Contrastive vision-language models like CLIP have gained popularity for their
versatile applicable learned representations in various downstream tasks.
Despite their successes in some tasks, like zero-shot image recognition, they
also perform surprisingly poor on other tasks, like attribute detection.
Previous work has attributed these challenges to the modality gap, a separation
of image and text in the shared representation space, and a bias towards
objects over other factors, such as attributes. In this work we investigate
both phenomena. We find that only a few embedding dimensions drive the modality
gap. Further, we propose a measure for object bias and find that object bias
does not lead to worse performance on other concepts, such as attributes. But
what leads to the emergence of the modality gap and object bias? To answer this
question we carefully designed an experimental setting which allows us to
control the amount of shared information between the modalities. This revealed
that the driving factor behind both, the modality gap and the object bias, is
the information imbalance between images and captions.

鉴于多模态差异，对象偏差及信息不平衡等因素，本研究通过对比视觉语言模型进行深入调查并提出了量化对象偏差的方法，揭示了信息不平衡是产生多模态差异和对象偏差的驱动因素。

两种效应，一个触发器：关于对比式视觉语言表示学习中的模态差异、物体偏见和信息不平衡

Two Effects, One Trigger: On the Modality Gap, Object Bias, and  Information Imbalance in Contrastive Vision-Language Representation Learning

Vision and Language (VL) models offer an effective method for aligning
representation spaces of images and text, leading to numerous applications such
as cross-modal retrieval, visual question answering, captioning, and more.
However, the aligned image-text spaces learned by all the popular VL models are
still suffering from the so-called `object bias' - their representations behave
as `bags of nouns', mostly ignoring or downsizing the attributes, relations,
and states of objects described/appearing in texts/images. Although some great
attempts at fixing these `compositional reasoning' issues were proposed in the
recent literature, the problem is still far from being solved. In this paper,
we uncover two factors limiting the VL models' compositional reasoning
performance. These two factors are properties of the paired VL dataset used for
finetuning and pre-training the VL model: (i) the caption quality, or in other
words `image-alignment', of the texts; and (ii) the `density' of the captions
in the sense of mentioning all the details appearing on the image. We propose a
fine-tuning approach for automatically treating these factors leveraging a
standard VL dataset (CC3M). Applied to CLIP, we demonstrate its significant
compositional reasoning performance increase of up to $\sim27\%$ over the base
model, up to $\sim20\%$ over the strongest baseline, and by $6.7\%$ on average.

本文探索了影响视觉语言模型组合推理性能的两个因素：对齐图像 - 文本数据集的质量和文本的密度。通过在 CC3M 数据集上 fine-tuning CLIP 模型，我们证明了自动处理这些影响因素的方法可以使模型的组合推理性能显著提高，基于基线模型的表现增加了约 20％，平均提高 6.7％。