With advances in the quality of text-to-image (T2I) models has come interest
in benchmarking their prompt faithfulness-the semantic coherence of generated
images to the prompts they were conditioned on. A variety of T2I faithfulness
metrics have been proposed, leveraging advances in cross-modal embeddings and
vision-language models (VLMs). However, these metrics are not rigorously
compared and benchmarked, instead presented against few weak baselines by
correlation to human Likert scores over a set of easy-to-discriminate images.
We introduce T2IScoreScore (TS2), a curated set of semantic error graphs
containing a prompt and a set increasingly erroneous images. These allow us to
rigorously judge whether a given prompt faithfulness metric can correctly order
images with respect to their objective error count and significantly
discriminate between different error nodes, using meta-metric scores derived
from established statistical tests. Surprisingly, we find that the
state-of-the-art VLM-based metrics (e.g., TIFA, DSG, LLMScore, VIEScore) we
tested fail to significantly outperform simple feature-based metrics like
CLIPScore, particularly on a hard subset of naturally-occurring T2I model
errors. TS2 will enable the development of better T2I prompt faithfulness
metrics through more rigorous comparison of their conformity to expected
orderings and separations under objective criteria.

对文本到图像模型中的提示准确性进行基准测试，比较评估了多种提示信实度度量，发现当前的视觉语言模型基于向量的度量无法在许多困难的 T2I 模型错误情况下显着优于基于特征的度量。介绍了一套语义错误图形，TS2，用于严格判断给定的提示信实度度量指标是否能够正确地对图像进行排序和区分不同的错误节点，以期通过客观的准则更严格地比较和发展更好的 T2I 期望提示信实度度量指标。