Existing text-to-image (T2I) diffusion models usually struggle in
interpreting complex prompts, especially those with quantity, object-attribute
binding, and multi-subject descriptions. In this work, we introduce a semantic
panel as the middleware in decoding texts to images, supporting the generator
to better follow instructions. The panel is obtained through arranging the
visual concepts parsed from the input text by the aid of large language models,
and then injected into the denoising network as a detailed control signal to
complement the text condition. To facilitate text-to-panel learning, we come up
with a carefully designed semantic formatting protocol, accompanied by a
fully-automatic data preparation pipeline. Thanks to such a design, our
approach, which we call Ranni, manages to enhance a pre-trained T2I generator
regarding its textual controllability. More importantly, the introduction of
the generative middleware brings a more convenient form of interaction (i.e.,
directly adjusting the elements in the panel or using language instructions)
and further allows users to finely customize their generation, based on which
we develop a practical system and showcase its potential in continuous
generation and chatting-based editing.

我们引入了一种语义面板作为中介来解码文本到图像的过程，用于支持生成器更好地遵循指示。这种面板是通过将从输入文本解析的视觉概念经由大型语言模型的辅助排列而成，然后注入到去噪网络作为详细的控制信号来补充文本条件。通过这样的设计，我们的方法 Ranni 在提高预训练的 T2I 生成器的文本可控性方面取得了进展，并引入了生成性中介，提供了一种更方便的交互形式，并允许用户根据其进行精细的生成定制，基于此我们开发了一个实用系统，并展示了其在连续生成和基于聊天的编辑中的潜力。