Text-to-image diffusion models achieved a remarkable leap in capabilities
over the last few years, enabling high-quality and diverse synthesis of images
from a textual prompt. However, even the most advanced models often struggle to
precisely follow all of the directions in their prompts. The vast majority of
these models are trained on datasets consisting of (image, caption) pairs where
the images often come from the web, and the captions are their HTML alternate
text. A notable example is the LAION dataset, used by Stable Diffusion and
other models. In this work we observe that these captions are often of low
quality, and argue that this significantly affects the model's capability to
understand nuanced semantics in the textual prompts. We show that by relabeling
the corpus with a specialized automatic captioning model and training a
text-to-image model on the recaptioned dataset, the model benefits
substantially across the board. First, in overall image quality: e.g. FID 14.84
vs. the baseline of 17.87, and 64.3% improvement in faithful image generation
according to human evaluation. Second, in semantic alignment, e.g. semantic
object accuracy 84.34 vs. 78.90, counting alignment errors 1.32 vs. 1.44 and
positional alignment 62.42 vs. 57.60. We analyze various ways to relabel the
corpus and provide evidence that this technique, which we call RECAP, both
reduces the train-inference discrepancy and provides the model with more
information per example, increasing sample efficiency and allowing the model to
better understand the relations between captions and images.

通过重新标注语料库并以此为基础训练文本到图像模型，可以显著提高模型的图像质量和语义对齐，并减少训练与推理之间的差异，增加样例效率，使模型更好地理解标题和图像之间的关系。