Performant vision-language (VL) models like CLIP represent captions using a
single vector. How much information about language is lost in this bottleneck?
We first curate CompPrompts, a set of increasingly compositional image captions
that VL models should be able to capture (e.g., single object, to
object+property, to multiple interacting objects). Then, we train text-only
recovery probes that aim to reconstruct captions from single-vector text
representations produced by several VL models. This approach doesn't require
images, allowing us to test on a broader range of scenes compared to prior
work. We find that: 1) CLIP's text encoder falls short on object relationships,
attribute-object association, counting, and negations; 2) some text encoders
work significantly better than others; and 3) text-only recovery performance
predicts multi-modal matching performance on ControlledImCaps: a new evaluation
benchmark we collect+release consisting of fine-grained compositional
images+captions. Specifically -- our results suggest text-only recoverability
is a necessary (but not sufficient) condition for modeling compositional
factors in contrastive vision+language models. We release data+code.

通过创建 CompPrompts 数据集的方法，研究了视觉 - 语言模型的编码模式对语言信息的损失情况，并提出了基于文本恢复的方法和 ControlledImCaps 评估基准，发现文本恢复能力与模型适用于呈现目标属性关系、计数、否定和多目标交互等复合要素的能力相关性强，并表明这是对偶视觉 + 语音模型的必要条件。