Recently, text-to-image diffusion models have demonstrated impressive ability
to generate high-quality images conditioned on the textual input. However,
these models struggle to accurately adhere to textual instructions regarding
spatial layout information. While previous research has primarily focused on
aligning cross-attention maps with layout conditions, they overlook the impact
of the initialization noise on the layout guidance. To achieve better layout
control, we propose leveraging a spatial-aware initialization noise during the
denoising process. Specifically, we find that the inverted reference image with
finite inversion steps contains valuable spatial awareness regarding the
object's position, resulting in similar layouts in the generated images. Based
on this observation, we develop an open-vocabulary framework to customize a
spatial-aware initialization noise for each layout condition. Without modifying
other modules except the initialization noise, our approach can be seamlessly
integrated as a plug-and-play module within other training-free layout guidance
frameworks. We evaluate our approach quantitatively and qualitatively on the
available Stable Diffusion model and COCO dataset. Equipped with the
spatial-aware latent initialization, our method significantly improves the
effectiveness of layout guidance while preserving high-quality content.

通过利用空间感知的初始化噪声，我们提出了一种自定义空间感知初始化噪声的开放词汇框架，以有效提高布局引导的效果。

针对可控图像生成的空间感知潜变量初始化

Spatial-Aware Latent Initialization for Controllable Image Generation

Text-to-image (T2I) generation has seen significant growth over the past few
years. Despite this, there has been little work on generating diagrams with T2I
models. A diagram is a symbolic/schematic representation that explains
information using structurally rich and spatially complex visualizations (e.g.,
a dense combination of related objects, text labels, directional arrows,
connection lines, etc.). Existing state-of-the-art T2I models often fail at
diagram generation because they lack fine-grained object layout control when
many objects are densely connected via complex relations such as arrows/lines
and also often fail to render comprehensible text labels. To address this gap,
we present DiagrammerGPT, a novel two-stage text-to-diagram generation
framework that leverages the layout guidance capabilities of LLMs (e.g., GPT-4)
to generate more accurate open-domain, open-platform diagrams. In the first
stage, we use LLMs to generate and iteratively refine 'diagram plans' (in a
planner-auditor feedback loop) which describe all the entities (objects and
text labels), their relationships (arrows or lines), and their bounding box
layouts. In the second stage, we use a diagram generator, DiagramGLIGEN, and a
text label rendering module to generate diagrams following the diagram plans.
To benchmark the text-to-diagram generation task, we introduce AI2D-Caption, a
densely annotated diagram dataset built on top of the AI2D dataset. We show
quantitatively and qualitatively that our DiagrammerGPT framework produces more
accurate diagrams, outperforming existing T2I models. We also provide
comprehensive analysis including open-domain diagram generation, vector graphic
diagram generation in different platforms, human-in-the-loop diagram plan
editing, and multimodal planner/auditor LLMs (e.g., GPT-4Vision). We hope our
work can inspire further research on diagram generation via T2I models and
LLMs.

提出了 DiagrammerGPT，它是一个新颖的两阶段文本生成图表的框架，利用了 LLMs 的布局指导能力来生成更准确的开放领域、开放平台图表。