Large-scale models combining text and images have made incredible progress in
recent years. However, they can still fail at tasks requiring compositional
knowledge, such as correctly picking out a red cube from a picture of multiple
shapes. We examine the ability of CLIP (Radford et al., 2021), to caption
images requiring compositional knowledge. We implement five compositional
language models to probe the kinds of structure that CLIP may be using, and
develop a novel training algorithm, Compositional Skipgram for Images (CoSI),
to train these models. We look at performance in attribute-based tasks,
requiring the identification of a particular combination of attribute and
object (such as "red cube"), and in relational settings, where the spatial
relation between two shapes (such as "cube behind sphere") must be identified.
We find that in some conditions, CLIP is able to learn attribute-object
labellings, and to generalize to unseen attribute-object combinations. However,
we also see evidence that CLIP is not able to bind features together reliably.
Moreover, CLIP is not able to reliably learn relations between objects, whereas
some compositional models are able to learn these perfectly. Of the five models
we developed, none were able to generalize to unseen relations.

本文探讨了 CLIP 在合成知识下图像描述的能力，提出了五种计算模型并设计出新的训练算法 CoSI，在属性 - 对象标记任务和空间关系任务中测试了 CLIP 的表现，结果表明虽然 CLIP 在属性 - 对象标记任务中表现良好且能够推广至新的未知属性 - 对象组合，但其无法可靠地绑定特性，对于对象之间的关系无法可靠地学习。