Progress in Text-to-Image (T2I) models has significantly improved the generation of images from textual descriptions. However, existing evaluation metrics do not adequately assess the models' ability to handle a diverse range of textual prompts, which is crucial for their generalizability. To address this, we introduce a new metric called Visual Language Evaluation Understudy (VLEU). VLEU uses large language models to sample from the visual text domain, the set of all possible input texts for T2I models, to generate a wide variety of prompts. The images generated from these prompts are evaluated based on their alignment with the input text using the CLIP model.VLEU quantifies a model's generalizability by computing the Kullback-Leibler divergence between the marginal distribution of the visual text and the conditional distribution of the images generated by the model. This metric provides a quantitative way to compare different T2I models and track improvements during model finetuning. Our experiments demonstrate the effectiveness of VLEU in evaluating the generalization capability of various T2I models, positioning it as an essential metric for future research in text-to-image synthesis.

本研究针对现有的文本到图像模型评估指标无法充分衡量模型处理多样文本提示能力的不足，提出了一种新指标VLEU。该指标通过计算视觉文本的边际分布与模型生成图像的条件分布之间的Kullback-Leibler散度，量化模型的普适性，实验结果表明VLEU对不同模型的评估有效，具有重要的研究意义。

VLEU：一种用于自动评估文本到图像模型普适性的方法