Most pre-trained learning systems are known to suffer from bias, which
typically emerges from the data, the model, or both. Measuring and quantifying
bias and its sources is a challenging task and has been extensively studied in
image captioning. Despite the significant effort in this direction, we observed
that existing metrics lack consistency in the inclusion of the visual signal.
In this paper, we introduce a new bias assessment metric, dubbed
$ImageCaptioner^2$, for image captioning. Instead of measuring the absolute
bias in the model or the data, $ImageCaptioner^2$ pay more attention to the
bias introduced by the model w.r.t the data bias, termed bias amplification.
Unlike the existing methods, which only evaluate the image captioning
algorithms based on the generated captions only, $ImageCaptioner^2$
incorporates the image while measuring the bias. In addition, we design a
formulation for measuring the bias of generated captions as prompt-based image
captioning instead of using language classifiers. Finally, we apply our
$ImageCaptioner^2$ metric across 11 different image captioning architectures on
three different datasets, i.e., MS-COCO caption dataset, Artemis V1, and
Artemis V2, and on three different protected attributes, i.e., gender, race,
and emotions. Consequently, we verify the effectiveness of our
$ImageCaptioner^2$ metric by proposing AnonymousBench, which is a novel human
evaluation paradigm for bias metrics. Our metric shows significant superiority
over the recent bias metric; LIC, in terms of human alignment, where the
correlation scores are 80% and 54% for our metric and LIC, respectively. The
code is available at this https URL

本文提出了一种新的评估图像字幕偏见的量化评估度量 ImageCaptioner2，通过对生成的字幕进行基于提示的图像字幕评估，以评估基于图像的字幕算法的偏见，通过在 MS-COCO 字幕数据集，Artemis V1 和 Artemis V2 上对 11 种不同的基于图像的字幕结构以及三种不同的受保护属性（性别，种族和情绪）进行测试来证实该度量的有效性，并验证了我们的度量与最近的度量（LIC）相比，具有更高的人类一致性。