Text-to-image (T2I) models have recently experienced rapid development,
achieving astonishing performance in terms of fidelity and textual alignment
capabilities. However, given a long paragraph (up to 512 words), these
generation models still struggle to achieve strong alignment and are unable to
generate images depicting complex scenes. In this paper, we introduce an
information-enriched diffusion model for paragraph-to-image generation task,
termed ParaDiffusion, which delves into the transference of the extensive
semantic comprehension capabilities of large language models to the task of
image generation. At its core is using a large language model (e.g., Llama V2)
to encode long-form text, followed by fine-tuning with LORA to alignthe
text-image feature spaces in the generation task. To facilitate the training of
long-text semantic alignment, we also curated a high-quality paragraph-image
pair dataset, namely ParaImage. This dataset contains a small amount of
high-quality, meticulously annotated data, and a large-scale synthetic dataset
with long text descriptions being generated using a vision-language model.
Experiments demonstrate that ParaDiffusion outperforms state-of-the-art models
(SD XL, DeepFloyd IF) on ViLG-300 and ParaPrompts, achieving up to 15% and 45%
human voting rate improvements for visual appeal and text faithfulness,
respectively. The code and dataset will be released to foster community
research on long-text alignment.

介绍了一种信息丰富的扩散模型，名为 ParaDiffusion，用于段落到图像生成任务，通过利用大型语言模型来提升图像生成模型的语义理解能力，并在长文本语义对齐训练方面取得了优异结果。