The combination of strong visual backbones and Large Language Model (LLM)
reasoning has led to Large Multimodal Models (LMMs) becoming the current
standard for a wide range of vision and language (VL) tasks. However, recent
research has shown that even the most advanced LMMs still struggle to capture
aspects of compositional visual reasoning, such as attributes and relationships
between objects. One solution is to utilize scene graphs (SGs)--a formalization
of objects and their relations and attributes that has been extensively used as
a bridge between the visual and textual domains. Yet, scene graph data requires
scene graph annotations, which are expensive to collect and thus not easily
scalable. Moreover, finetuning an LMM based on SG data can lead to catastrophic
forgetting of the pretraining objective. To overcome this, inspired by
chain-of-thought methods, we propose Compositional Chain-of-Thought (CCoT), a
novel zero-shot Chain-of-Thought prompting method that utilizes SG
representations in order to extract compositional knowledge from an LMM.
Specifically, we first generate an SG using the LMM, and then use that SG in
the prompt to produce a response. Through extensive experiments, we find that
the proposed CCoT approach not only improves LMM performance on several vision
and language VL compositional benchmarks but also improves the performance of
several popular LMMs on general multimodal benchmarks, without the need for
fine-tuning or annotated ground-truth SGs.

本研究提出了一种基于场景图的链式思维（Compositional Chain-of-Thought，CCoT）方法，通过在大型多模态模型（LMM）中利用场景图表示来提取组成性知识，从而改进了多种视觉语言（VL）组成性基准的性能以及多模态基准的性能。

大型多模态模型的构图推理引导

Compositional Chain-of-Thought Prompting for Large Multimodal Models

Large-scale models combining text and images have made incredible progress in
recent years. However, they can still fail at tasks requiring compositional
knowledge, such as correctly picking out a red cube from a picture of multiple
shapes. We examine the ability of CLIP (Radford et al., 2021), to caption
images requiring compositional knowledge. We implement five compositional
language models to probe the kinds of structure that CLIP may be using, and
develop a novel training algorithm, Compositional Skipgram for Images (CoSI),
to train these models. We look at performance in attribute-based tasks,
requiring the identification of a particular combination of attribute and
object (such as "red cube"), and in relational settings, where the spatial
relation between two shapes (such as "cube behind sphere") must be identified.
We find that in some conditions, CLIP is able to learn attribute-object
labellings, and to generalize to unseen attribute-object combinations. However,
we also see evidence that CLIP is not able to bind features together reliably.
Moreover, CLIP is not able to reliably learn relations between objects, whereas
some compositional models are able to learn these perfectly. Of the five models
we developed, none were able to generalize to unseen relations.

本文探讨了 CLIP 在合成知识下图像描述的能力，提出了五种计算模型并设计出新的训练算法 CoSI，在属性 - 对象标记任务和空间关系任务中测试了 CLIP 的表现，结果表明虽然 CLIP 在属性 - 对象标记任务中表现良好且能够推广至新的未知属性 - 对象组合，但其无法可靠地绑定特性，对于对象之间的关系无法可靠地学习。