Large-scale text-to-image diffusion models have significantly improved the
state of the art in generative image modelling and allow for an intuitive and
powerful user interface to drive the image generation process. Expressing
spatial constraints, e.g. to position specific objects in particular locations,
is cumbersome using text; and current text-based image generation models are
not able to accurately follow such instructions. In this paper we consider
image generation from text associated with segments on the image canvas, which
combines an intuitive natural language interface with precise spatial control
over the generated content. We propose ZestGuide, a zero-shot segmentation
guidance approach that can be plugged into pre-trained text-to-image diffusion
models, and does not require any additional training. It leverages implicit
segmentation maps that can be extracted from cross-attention layers, and uses
them to align the generation with input masks. Our experimental results combine
high image quality with accurate alignment of generated content with input
segmentations, and improve over prior work both quantitatively and
qualitatively, including methods that require training on images with
corresponding segmentations. Compared to Paint with Words, the previous
state-of-the art in image generation with zero-shot segmentation conditioning,
we improve by 5 to 10 mIoU points on the COCO dataset with similar FID scores.

使用文本生成图像的方法已显著提高了生成图像建模的状态，通过结合自然语言界面和空间控制，在本文中，我们考虑了与图像画布上特定位置对象相关的文本生成图像的问题，并提出了 ZestGuide，这是一种零 - shot 分割指导方法，可插入预先训练的文本到图像扩散模型中，我们的实验结果表明，在精准对齐所需的输入分割的情况下，与使用相应分割进行训练的组件相比，提高了图像质量，而在 COCO 数据集上，我们的结果比 Paint with Words 更好。