Benchmarks of the multilingual capabilities of text-to-image (T2I) models
compare generated images prompted in a test language to an expected image
distribution over a concept set. One such benchmark, "Conceptual Coverage
Across Languages" (CoCo-CroLa), assesses the tangible noun inventory of T2I
models by prompting them to generate pictures from a concept list translated to
seven languages and comparing the output image populations. Unfortunately, we
find that this benchmark contains translation errors of varying severity in
Spanish, Japanese, and Chinese. We provide corrections for these errors and
analyze how impactful they are on the utility and validity of CoCo-CroLa as a
benchmark. We reassess multiple baseline T2I models with the revisions, compare
the outputs elicited under the new translations to those conditioned on the
old, and show that a correction's impactfulness on the image-domain benchmark
results can be predicted in the text domain with similarity scores. Our
findings will guide the future development of T2I multilinguality metrics by
providing analytical tools for practical translation decisions.

基于图像的多语言（T2I）模型能力的基准测试比较生成的图像与期望图像分布的语言特定概念集合，其中一个名为 CoCo-CroLa 的基准测试通过将概念列表翻译成七种语言并比较输出图像群体来评估 T2I 模型的有形名词库存。不幸的是，我们发现该基准测试在西班牙语、日语和中文中包含不同严重程度的翻译错误。我们提供了这些错误的更正，并分析这些错误对 CoCo-CroLa 作为基准测试的实用性和有效性的影响。我们使用相似性评分在文本领域预测了纠正对图像领域基准测试结果的影响，并重新评估了多个基准 T2I 模型的输出，将新翻译的输出与旧翻译的输出进行了比较。我们的发现将为 T2I 多语言度量的未来发展提供分析工具，以进行实际的翻译决策。