One challenge in text-to-image (T2I) generation is the inadvertent reflection
of culture gaps present in the training data, which signifies the disparity in
generated image quality when the cultural elements of the input text are rarely
collected in the training set. Although various T2I models have shown
impressive but arbitrary examples, there is no benchmark to systematically
evaluate a T2I model's ability to generate cross-cultural images. To bridge the
gap, we propose a Challenging Cross-Cultural (C3) benchmark with comprehensive
evaluation criteria, which can assess how well-suited a model is to a target
culture. By analyzing the flawed images generated by the Stable Diffusion model
on the C3 benchmark, we find that the model often fails to generate certain
cultural objects. Accordingly, we propose a novel multi-modal metric that
considers object-text alignment to filter the fine-tuning data in the target
culture, which is used to fine-tune a T2I model to improve cross-cultural
generation. Experimental results show that our multi-modal metric provides
stronger data selection performance on the C3 benchmark than existing metrics,
in which the object-text alignment is crucial. We release the benchmark, data,
code, and generated images to facilitate future research on culturally diverse
T2I generation (this https URL).

文本到图像（T2I）生成中的一个挑战是训练数据中存在的文化差距意外反映，当输入文本的文化要素在训练集中很少收集时，这意味着生成的图像质量的不一致性；为了弥合差距，我们提出了一个全面评估标准的具有挑战性的跨文化（C3）基准，用于评估模型生成跨文化图像的适应性；通过分析 C3 基准上 Stable Diffusion 模型生成的有缺陷图像，我们发现模型常常无法生成某些文化对象；因此，我们提出一种新颖的多模态度量，考虑了对象与文本的对齐，用于筛选目标文化中的微调数据，以改进跨文化生成的 T2I 模型；实验结果表明，我们的多模态度量在 C3 基准上提供了比现有度量更强的数据选择性能，其中对象和文本的对齐至关重要；我们发布了基准，数据，代码和生成的图像，以促进未来在文化多样化的 T2I 生成方面的研究。