Existing text-to-image models struggle to follow complex text prompts,
raising the need for extra grounding inputs for better controllability. In this
work, we propose to decompose a scene into visual primitives - denoted as dense
blob representations - that contain fine-grained details of the scene while
being modular, human-interpretable, and easy-to-construct. Based on blob
representations, we develop a blob-grounded text-to-image diffusion model,
termed BlobGEN, for compositional generation. Particularly, we introduce a new
masked cross-attention module to disentangle the fusion between blob
representations and visual features. To leverage the compositionality of large
language models (LLMs), we introduce a new in-context learning approach to
generate blob representations from text prompts. Our extensive experiments show
that BlobGEN achieves superior zero-shot generation quality and better
layout-guided controllability on MS-COCO. When augmented by LLMs, our method
exhibits superior numerical and spatial correctness on compositional image
generation benchmarks. Project page: this https URL

现有的文本 - 图像模型在遵循复杂文本提示上困难重重，因此需要额外的基础输入以提高可控性。本研究提出将场景分解为可容纳细粒度细节、模块化、可解释的、易于构建的视觉基元 - 密集 Blob 表示。基于 Blob 表示，我们开发了一种基于 Blob 的文本 - 图像扩散模型 BlobGEN，用于组合生成。通过引入新的屏蔽式交叉注意力模块来解开 Blob 表示和视觉特征之间的融合，以发挥大型语言模型 (LLMs) 的组合性。我们引入了一种新的上下文学习方法来从文本提示生成 Blob 表示。我们广泛的实验表明，BlobGEN 在 MS-COCO 上实现了卓越的零样本生成质量和更好的布局引导可控性。当与 LLMs 结合使用时，我们的方法在组合图像生成基准上展现出卓越的数值和空间正确性。

密集斑点表示的组合式文本到图像生成

Compositional Text-to-Image Generation with Dense Blob Representations

In compositional zero-shot learning, the goal is to recognize unseen
compositions (e.g. old dog) of observed visual primitives states (e.g. old,
cute) and objects (e.g. car, dog) in the training set. This is challenging
because the same state can for example alter the visual appearance of a dog
drastically differently from a car. As a solution, we propose a novel graph
formulation called Compositional Graph Embedding (CGE) that learns image
features, compositional classifiers, and latent representations of visual
primitives in an end-to-end manner. The key to our approach is exploiting the
dependency between states, objects, and their compositions within a graph
structure to enforce the relevant knowledge transfer from seen to unseen
compositions. By learning a joint compatibility that encodes semantics between
concepts, our model allows for generalization to unseen compositions without
relying on an external knowledge base like WordNet. We show that in the
challenging generalized compositional zero-shot setting our CGE significantly
outperforms the state of the art on MIT-States and UT-Zappos. We also propose a
new benchmark for this task based on the recent GQA dataset. Code is available
at: this https URL

本研究提出了一种名为 Compositional Graph Embedding（CGE）的图形学习方法，可以实现对已出现的视觉对象和视觉基元状态的组合的识别和理解，进而实现了基于组合的零样本学习，避免了外部知识库的依赖。CGE 不仅可以学习图像特征，合成分类器，还可以学习视觉基元的潜在特征，最终可以进行对未出现过的组合图像识别，相比其他方法，CGE 在 MIT-States 和 UT-Zappos 等数据集上具有更好的性能表现。

学习图嵌入实现组合式零样本学习

Learning Graph Embeddings for Compositional Zero-shot Learning

We introduce a novel method for representation learning that uses an
artificial supervision signal based on counting visual primitives. This
supervision signal is obtained from an equivariance relation, which does not
require any manual annotation. We relate transformations of images to
transformations of the representations. More specifically, we look for the
representation that satisfies such relation rather than the transformations
that match a given representation. In this paper, we use two image
transformations in the context of counting: scaling and tiling. The first
transformation exploits the fact that the number of visual primitives should be
invariant to scale. The second transformation allows us to equate the total
number of visual primitives in each tile to that in the whole image. These two
transformations are combined in one constraint and used to train a neural
network with a contrastive loss. The proposed task produces representations
that perform on par or exceed the state of the art in transfer learning
benchmarks.

本文介绍了一种基于视觉原语计数的人工监督信号的表示学习新方法，使用无需手动注释的等变关系获取监督信号，并使用两个图像变换（缩放和平铺）组合成一个约束条件，用对比损失训练神经网络，以产生在迁移学习基准测试中与当前最新技术相媲美或超越其的表示。