Existing work has observed that current text-to-image systems do not
accurately reflect explicit spatial relations between objects such as 'left of'
or 'below'. We hypothesize that this is because explicit spatial relations
rarely appear in the image captions used to train these models. We propose an
automatic method that, given existing images, generates synthetic captions that
contain 14 explicit spatial relations. We introduce the Spatial Relation for
Generation (SR4G) dataset, which contains 9.9 millions image-caption pairs for
training, and more than 60 thousand captions for evaluation. In order to test
generalization we also provide an 'unseen' split, where the set of objects in
the train and test captions are disjoint. SR4G is the first dataset that can be
used to spatially fine-tune text-to-image systems. We show that fine-tuning two
different Stable Diffusion models (denoted as SD$_{SR4G}$) yields up to 9
points improvements in the VISOR metric. The improvement holds in the 'unseen'
split, showing that SD$_{SR4G}$ is able to generalize to unseen objects.
SD$_{SR4G}$ improves the state-of-the-art with fewer parameters, and avoids
complex architectures. Our analysis shows that improvement is consistent for
all relations. The dataset and the code will be publicly available.

现有文献观察到当前的文本到图像系统不能准确反映物体之间的显式空间关系，我们假设这是因为训练这些模型的图像标题很少包含显式空间关系。为此，我们提出了一种自动方法，给定现有图像生成包含 14 种显式空间关系的合成标题。我们引入了 Spatial Relation for Generation (SR4G) 数据集，其中包含 990 万个图像标题对进行训练，并且包含超过 6 万个用于评估的标题。为了测试泛化性能，我们还提供了一个 ' 未见过 ' 的分割，其中训练和测试标题中的对象集是不相交的。SR4G 是首个可以用于空间微调文本到图像系统的数据集。我们显示微调两个不同的稳定扩散模型（称为 SD$_{SR4G}$）可以使 VISOR 指标提升高达 9 个点。改进结果在 ' 未见过 ' 的分割中依然有效，表明 SD$_{SR4G}$ 可以推广到未见过的对象。SD$_{SR4G}$ 通过更少的参数改进了最先进的方法，并避免了复杂的架构。我们的分析显示改进对于所有关系都是一致的。数据集和代码将公开提供。

通过自动派生的数据集改进文本到图像生成中的显式空间关系

Improving Explicit Spatial Relationships in Text-to-Image Generation  through an Automatically Derived Dataset

Diffusion models have opened up new avenues for the field of image
generation, resulting in the proliferation of high-quality models shared on
open-source platforms. However, a major challenge persists in current
text-to-image systems are often unable to handle diverse inputs, or are limited
to single model results. Current unified attempts often fall into two
orthogonal aspects: i) parse Diverse Prompts in input stage; ii) activate
expert model to output. To combine the best of both worlds, we propose
DiffusionGPT, which leverages Large Language Models (LLM) to offer a unified
generation system capable of seamlessly accommodating various types of prompts
and integrating domain-expert models. DiffusionGPT constructs domain-specific
Trees for various generative models based on prior knowledge. When provided
with an input, the LLM parses the prompt and employs the Trees-of-Thought to
guide the selection of an appropriate model, thereby relaxing input constraints
and ensuring exceptional performance across diverse domains. Moreover, we
introduce Advantage Databases, where the Tree-of-Thought is enriched with human
feedback, aligning the model selection process with human preferences. Through
extensive experiments and comparisons, we demonstrate the effectiveness of
DiffusionGPT, showcasing its potential for pushing the boundaries of image
synthesis in diverse domains.

DiffusionGPT 结合了扩散模型、文本到图像系统和领域特定树，提供了一个统一的生成系统，能够适应各种类型的提示并集成领域专家模型，推动了多领域图像合成的边界。

DiffusionGPT: 基于 LLM 的文本到图像生成系统

DiffusionGPT: LLM-Driven Text-to-Image Generation System

Recent progress in text-to-image (TTI) systems, such as StableDiffusion,
Imagen, and DALL-E 2, have made it possible to create realistic images with
simple text prompts. It is tempting to use these systems to eliminate the
manual task of obtaining natural images for training a new machine learning
classifier. However, in all of the experiments performed to date, classifiers
trained solely with synthetic images perform poorly at inference, despite the
images used for training appearing realistic. Examining this apparent
incongruity in detail gives insight into the limitations of the underlying
image generation processes. Through the lens of diversity in image creation
vs.accuracy of what is created, we dissect the differences in semantic
mismatches in what is modeled in synthetic vs. natural images. This will
elucidate the roles of the image-languag emodel, CLIP, and the image generation
model, diffusion. We find four issues that limit the usefulness of TTI systems
for this task: ambiguity, adherence to prompt, lack of diversity, and inability
to represent the underlying concept. We further present surprising insights
into the geometry of CLIP embeddings.

通过对文本到图像系统的进展和合成图像在训练和推理过程中的不足进行研究，揭示了语义不匹配、多样性缺乏和表征基础概念的无能等问题，同时提出了对 CLIP 嵌入几何的令人惊讶的见解。