Benchmarks of the multilingual capabilities of text-to-image (T2I) models
compare generated images prompted in a test language to an expected image
distribution over a concept set. One such benchmark, "Conceptual Coverage
Across Languages" (CoCo-CroLa), assesses the tangible noun inventory of T2I
models by prompting them to generate pictures from a concept list translated to
seven languages and comparing the output image populations. Unfortunately, we
find that this benchmark contains translation errors of varying severity in
Spanish, Japanese, and Chinese. We provide corrections for these errors and
analyze how impactful they are on the utility and validity of CoCo-CroLa as a
benchmark. We reassess multiple baseline T2I models with the revisions, compare
the outputs elicited under the new translations to those conditioned on the
old, and show that a correction's impactfulness on the image-domain benchmark
results can be predicted in the text domain with similarity scores. Our
findings will guide the future development of T2I multilinguality metrics by
providing analytical tools for practical translation decisions.

基于图像的多语言（T2I）模型能力的基准测试比较生成的图像与期望图像分布的语言特定概念集合，其中一个名为 CoCo-CroLa 的基准测试通过将概念列表翻译成七种语言并比较输出图像群体来评估 T2I 模型的有形名词库存。不幸的是，我们发现该基准测试在西班牙语、日语和中文中包含不同严重程度的翻译错误。我们提供了这些错误的更正，并分析这些错误对 CoCo-CroLa 作为基准测试的实用性和有效性的影响。我们使用相似性评分在文本领域预测了纠正对图像领域基准测试结果的影响，并重新评估了多个基准 T2I 模型的输出，将新翻译的输出与旧翻译的输出进行了比较。我们的发现将为 T2I 多语言度量的未来发展提供分析工具，以进行实际的翻译决策。

在多语言概念上进行文本到图像模型公平评估的翻译错误和挑战

Lost in Translation? Translation Errors and Challenges for Fair  Assessment of Text-to-Image Models on Multilingual Concepts

We propose "Conceptual Coverage Across Languages" (CoCo-CroLa), a technique
for benchmarking the degree to which any generative text-to-image system
provides multilingual parity to its training language in terms of tangible
nouns. For each model we can assess "conceptual coverage" of a given target
language relative to a source language by comparing the population of images
generated for a series of tangible nouns in the source language to the
population of images generated for each noun under translation in the target
language. This technique allows us to estimate how well-suited a model is to a
target language as well as identify model-specific weaknesses, spurious
correlations, and biases without a-priori assumptions. We demonstrate how it
can be used to benchmark T2I models in terms of multilinguality, and how
despite its simplicity it is a good proxy for impressive generalization.

这篇论文介绍了一种名为 CoCo-CroLa 的技术，用于评估任何生成式文本到图像系统在有形名词方面提供多语言的程度，可以比较源语言和目标语言中为一系列有形名词生成的图像的数量，从而估计模型对目标语言的适用性，并识别模型特定的弱点、错误关联和偏见。