Rapid progress in text-to-image generative models coupled with their
deployment for visual content creation has magnified the importance of
thoroughly evaluating their performance and identifying potential biases. In
pursuit of models that generate images that are realistic, diverse, visually
appealing, and consistent with the given prompt, researchers and practitioners
often turn to automated metrics to facilitate scalable and cost-effective
performance profiling. However, commonly-used metrics often fail to account for
the full diversity of human preference; often even in-depth human evaluations
face challenges with subjectivity, especially as interpretations of evaluation
criteria vary across regions and cultures. In this work, we conduct a large,
cross-cultural study to study how much annotators in Africa, Europe, and
Southeast Asia vary in their perception of geographic representation, visual
appeal, and consistency in real and generated images from state-of-the art
public APIs. We collect over 65,000 image annotations and 20 survey responses.
We contrast human annotations with common automated metrics, finding that human
preferences vary notably across geographic location and that current metrics do
not fully account for this diversity. For example, annotators in different
locations often disagree on whether exaggerated, stereotypical depictions of a
region are considered geographically representative. In addition, the utility
of automatic evaluations is dependent on assumptions about their set-up, such
as the alignment of feature extractors with human perception of object
similarity or the definition of "appeal" captured in reference datasets used to
ground evaluations. We recommend steps for improved automatic and human
evaluations.

通过进行大规模的跨文化研究，我们发现不同地理位置的人对地理表达、视觉吸引力和一致性的偏好存在显著差异，并且目前常用的自动评估指标不能充分考虑这种多样性。我们建议改进自动和人工评估方法。