Large-scale models combining text and images have made incredible progress in
recent years. However, they can still fail at tasks requiring compositional
knowledge, such as correctly picking out a red cube from a picture of multiple
shapes. We examine the ability of CLIP (Radford et al., 2021), to caption
images requiring compositional knowledge. We implement five compositional
language models to probe the kinds of structure that CLIP may be using, and
develop a novel training algorithm, Compositional Skipgram for Images (CoSI),
to train these models. We look at performance in attribute-based tasks,
requiring the identification of a particular combination of attribute and
object (such as "red cube"), and in relational settings, where the spatial
relation between two shapes (such as "cube behind sphere") must be identified.
We find that in some conditions, CLIP is able to learn attribute-object
labellings, and to generalize to unseen attribute-object combinations. However,
we also see evidence that CLIP is not able to bind features together reliably.
Moreover, CLIP is not able to reliably learn relations between objects, whereas
some compositional models are able to learn these perfectly. Of the five models
we developed, none were able to generalize to unseen relations.

本文探讨了 CLIP 在合成知识下图像描述的能力，提出了五种计算模型并设计出新的训练算法 CoSI，在属性 - 对象标记任务和空间关系任务中测试了 CLIP 的表现，结果表明虽然 CLIP 在属性 - 对象标记任务中表现良好且能够推广至新的未知属性 - 对象组合，但其无法可靠地绑定特性，对于对象之间的关系无法可靠地学习。

CLIP 是否绑定概念？探索大型图像模型的组合能力

Does CLIP Bind Concepts? Probing Compositionality in Large Image Models

Abstract object properties and their relations are deeply rooted in human
common sense, allowing people to predict the dynamics of the world even in
situations that are novel but governed by familiar laws of physics. Standard
machine learning models in model-based reinforcement learning are inadequate to
generalize in this way. Inspired by the classic framework of noisy
indeterministic deictic (NID) rules, we introduce here Neural NID, a method
that learns abstract object properties and relations between objects with a
suitably regularized graph neural network. We validate the greater
generalization capability of Neural NID on simple benchmarks specifically
designed to assess the transition dynamics learned by the model.

本研究提出了一种基于神经网络的嘈杂不确定的指示（Neural NID）算法，使用正则化的图神经网络学习抽象对象属性和对象之间的关系，并证实其具有更好的泛化性能。