Diffusion model based Text-to-Image has achieved impressive achievements
recently. Although current technology for synthesizing images is highly
advanced and capable of generating images with high fidelity, it is still
possible to give the show away when focusing on the text area in the generated
image. To address this issue, we introduce AnyText, a diffusion-based
multilingual visual text generation and editing model, that focuses on
rendering accurate and coherent text in the image. AnyText comprises a
diffusion pipeline with two primary elements: an auxiliary latent module and a
text embedding module. The former uses inputs like text glyph, position, and
masked image to generate latent features for text generation or editing. The
latter employs an OCR model for encoding stroke data as embeddings, which blend
with image caption embeddings from the tokenizer to generate texts that
seamlessly integrate with the background. We employed text-control diffusion
loss and text perceptual loss for training to further enhance writing accuracy.
AnyText can write characters in multiple languages, to the best of our
knowledge, this is the first work to address multilingual visual text
generation. It is worth mentioning that AnyText can be plugged into existing
diffusion models from the community for rendering or editing text accurately.
After conducting extensive evaluation experiments, our method has outperformed
all other approaches by a significant margin. Additionally, we contribute the
first large-scale multilingual text images dataset, AnyWord-3M, containing 3
million image-text pairs with OCR annotations in multiple languages. Based on
AnyWord-3M dataset, we propose AnyText-benchmark for the evaluation of visual
text generation accuracy and quality. Our project will be open-sourced on
this https URL to improve and promote the development of
text generation technology.

基于扩散模型的多语言视觉文本生成的 AnyText 模型，利用 OCR 模型和扩散管道生成准确和连贯的文本，通过文本控制扩散损失和文本感知损失进行训练来进一步增强书写准确性，并在评估实验证明我们的方法在视觉文本生成方面表现优异。