Vision and Language (VL) models offer an effective method for aligning
representation spaces of images and text, leading to numerous applications such
as cross-modal retrieval, visual question answering, captioning, and more.
However, the aligned image-text spaces learned by all the popular VL models are
still suffering from the so-called `object bias' - their representations behave
as `bags of nouns', mostly ignoring or downsizing the attributes, relations,
and states of objects described/appearing in texts/images. Although some great
attempts at fixing these `compositional reasoning' issues were proposed in the
recent literature, the problem is still far from being solved. In this paper,
we uncover two factors limiting the VL models' compositional reasoning
performance. These two factors are properties of the paired VL dataset used for
finetuning and pre-training the VL model: (i) the caption quality, or in other
words `image-alignment', of the texts; and (ii) the `density' of the captions
in the sense of mentioning all the details appearing on the image. We propose a
fine-tuning approach for automatically treating these factors leveraging a
standard VL dataset (CC3M). Applied to CLIP, we demonstrate its significant
compositional reasoning performance increase of up to $\sim27\%$ over the base
model, up to $\sim20\%$ over the strongest baseline, and by $6.7\%$ on average.

本文探索了影响视觉语言模型组合推理性能的两个因素：对齐图像 - 文本数据集的质量和文本的密度。通过在 CC3M 数据集上 fine-tuning CLIP 模型，我们证明了自动处理这些影响因素的方法可以使模型的组合推理性能显著提高，基于基线模型的表现增加了约 20％，平均提高 6.7％。