Large language models (LLMs) based on decoder-only transformers have
demonstrated superior text understanding capabilities compared to CLIP and
T5-series models. However, the paradigm for utilizing current advanced LLMs in
text-to-image diffusion models remains to be explored. We observed an unusual
phenomenon: directly using a large language model as the prompt encoder
significantly degrades the prompt-following ability in image generation. We
identified two main obstacles behind this issue. One is the misalignment
between the next token prediction training in LLM and the requirement for
discriminative prompt features in diffusion models. The other is the intrinsic
positional bias introduced by the decoder-only architecture. To deal with this
issue, we propose a novel framework to fully harness the capabilities of LLMs.
Through the carefully designed usage guidance, we effectively enhance the text
representation capability for prompt encoding and eliminate its inherent
positional bias. This allows us to integrate state-of-the-art LLMs into the
text-to-image generation model flexibly. Furthermore, we also provide an
effective manner to fuse multiple LLMs into our framework. Considering the
excellent performance and scaling capabilities demonstrated by the transformer
architecture, we further design an LLM-Infused Diffusion Transformer (LI-DiT)
based on the framework. We conduct extensive experiments to validate LI-DiT
across model size and data size. Benefiting from the inherent ability of the
LLMs and our innovative designs, the prompt understanding performance of LI-DiT
easily surpasses state-of-the-art open-source models as well as mainstream
closed-source commercial models including Stable Diffusion 3, DALL-E 3, and
Midjourney V6. The powerful LI-DiT-10B will be available after further
optimization and security checks.

利用新提出的框架，将大型语言模型与文本到图像生成模型灵活地集成在一起，并通过消除固有的位置偏差和增强文本表示能力提高预测性能。