Diffusion models have gained increasing attention for their impressive
generation abilities but currently struggle with rendering accurate and
coherent text. To address this issue, we introduce \textbf{TextDiffuser},
focusing on generating images with visually appealing text that is coherent
with backgrounds. TextDiffuser consists of two stages: first, a Transformer
model generates the layout of keywords extracted from text prompts, and then
diffusion models generate images conditioned on the text prompt and the
generated layout. Additionally, we contribute the first large-scale text images
dataset with OCR annotations, \textbf{MARIO-10M}, containing 10 million
image-text pairs with text recognition, detection, and character-level
segmentation annotations. We further collect the \textbf{MARIO-Eval} benchmark
to serve as a comprehensive tool for evaluating text rendering quality. Through
experiments and user studies, we show that TextDiffuser is flexible and
controllable to create high-quality text images using text prompts alone or
together with text template images, and conduct text inpainting to reconstruct
incomplete images with text. The code, model, and dataset will be available at
https://aka.ms/textdiffuser.

为解决扩散模型在渲染准确、连贯文本方面的问题，我们介绍了 TextDiffuser，并提供了第一个带有 OCR 标注的大规模图像文本数据集 MARIO-10M， 进行了实验和用户研究，表明 TextDiffuser 可以仅基于文本提示或与文本模板图像一起创建高质量的文本图像，并进行文本修复来重构不完整的带文本图像。完成的代码、模型和数据集可在官网 https://aka.ms/textdiffuser 获得。