Recent work has identified substantial disparities in generated images of
different geographic regions, including stereotypical depictions of everyday
objects like houses and cars. However, existing measures for these disparities
have been limited to either human evaluations, which are time-consuming and
costly, or automatic metrics evaluating full images, which are unable to
attribute these disparities to specific parts of the generated images. In this
work, we introduce a new set of metrics, Decomposed Indicators of Disparities
in Image Generation (Decomposed-DIG), that allows us to separately measure
geographic disparities in the depiction of objects and backgrounds in generated
images. Using Decomposed-DIG, we audit a widely used latent diffusion model and
find that generated images depict objects with better realism than backgrounds
and that backgrounds in generated images tend to contain larger regional
disparities than objects. We use Decomposed-DIG to pinpoint specific examples
of disparities, such as stereotypical background generation in Africa,
struggling to generate modern vehicles in Africa, and unrealistically placing
some objects in outdoor settings. Informed by our metric, we use a new
prompting structure that enables a 52% worst-region improvement and a 20%
average improvement in generated background diversity.

地理差异图像生成任务中的 Decomposed-DIG 指标通过量化对象和背景的差异，揭示了生成图像中地理上的差异。研究发现背景生成中的地理差异明显大于对象生成，并提出了一种新的提示结构以优化背景多样性。

地理差异在文本到图像模型中的分解评估

Decomposed evaluations of geographic disparities in text-to-image models

The unprecedented photorealistic results achieved by recent text-to-image
generative systems and their increasing use as plug-and-play content creation
solutions make it crucial to understand their potential biases. In this work,
we introduce three indicators to evaluate the realism, diversity and
prompt-generation consistency of text-to-image generative systems when prompted
to generate objects from across the world. Our indicators complement
qualitative analysis of the broader impact of such systems by enabling
automatic and efficient benchmarking of geographic disparities, an important
step towards building responsible visual content creation systems. We use our
proposed indicators to analyze potential geographic biases in state-of-the-art
visual content creation systems and find that: (1) models have less realism and
diversity of generations when prompting for Africa and West Asia than Europe,
(2) prompting with geographic information comes at a cost to prompt-consistency
and diversity of generated images, and (3) models exhibit more region-level
disparities for some objects than others. Perhaps most interestingly, our
indicators suggest that progress in image generation quality has come at the
cost of real-world geographic representation. Our comprehensive evaluation
constitutes a crucial step towards ensuring a positive experience of visual
content creation for everyone.

通过引入三个指标，我们评估了文本到图像生成系统在生成全球各地对象时的真实性、多样性和提示生成的一致性，分析了当前最先进的视觉内容生成系统中可能存在的地理偏见，并发现：（1）在提示生成非洲和西亚的对象时，模型的真实性和生成多样性较欧洲较低，（2）使用地理信息作为提示对生成的图片的一致性和多样性具有一定的负面影响，以及（3）在一些对象上，模型存在更多的区域级差异。引入这些指标是确保每个人在视觉内容创建中有积极体验的关键步骤。

评估图像生成中的地理多样性差异

DIG In: Evaluating Disparities in Image Generations with Indicators for  Geographic Diversity

For more than a decade, researchers have measured progress in object
recognition on ImageNet-based generalization benchmarks such as ImageNet-A, -C,
and -R. Recent advances in foundation models, trained on orders of magnitude
more data, have begun to saturate these standard benchmarks, but remain brittle
in practice. This suggests standard benchmarks, which tend to focus on
predefined or synthetic changes, may not be sufficient for measuring real world
generalization. Consequently, we propose studying generalization across
geography as a more realistic measure of progress using two datasets of objects
from households across the globe. We conduct an extensive empirical evaluation
of progress across nearly 100 vision models up to most recent foundation
models. We first identify a progress gap between standard benchmarks and
real-world, geographical shifts: progress on ImageNet results in up to 2.5x
more progress on standard generalization benchmarks than real-world
distribution shifts. Second, we study model generalization across geographies
by measuring the disparities in performance across regions, a more fine-grained
measure of real world generalization. We observe all models have large
geographic disparities, even foundation CLIP models, with differences of 7-20%
in accuracy between regions. Counter to modern intuition, we discover progress
on standard benchmarks fails to improve geographic disparities and often
exacerbates them: geographic disparities between the least performant models
and today's best models have more than tripled. Our results suggest scaling
alone is insufficient for consistent robustness to real-world distribution
shifts. Finally, we highlight in early experiments how simple last layer
retraining on more representative, curated data can complement scaling as a
promising direction of future work, reducing geographic disparity on both
benchmarks by over two-thirds.

通过对全球家庭物品的两个数据集进行广泛的实证评估，我们首先确定了标准基准和现实世界地理转变之间的进展差距，接着通过测量不同地区性能的差异，研究了模型在地理上的泛化能力，并发现了地理差异的增加是传统基准进展的副作用。最后，我们强调了在更具代表性的数据上重新训练简单的最后一层如何作为未来工作的有希望的方向，使得两个基准测试的地理差异减少了三分之二以上。