Considerable efforts to measure and mitigate gender bias in recent years have
led to the introduction of an abundance of tasks, datasets, and metrics used in
this vein. In this position paper, we assess the current paradigm of gender
bias evaluation and identify several flaws in it. First, we highlight the
importance of extrinsic bias metrics that measure how a model's performance on
some task is affected by gender, as opposed to intrinsic evaluations of model
representations, which are less strongly connected to specific harms to people
interacting with systems. We find that only a few extrinsic metrics are
measured in most studies, although more can be measured. Second, we find that
datasets and metrics are often coupled, and discuss how their coupling hinders
the ability to obtain reliable conclusions, and how one may decouple them. We
then investigate how the choice of the dataset and its composition, as well as
the choice of the metric, affect bias measurement, finding significant
variations across each of them. Finally, we propose several guidelines for more
reliable gender bias evaluation.

通过评估当前性别偏见评估范式并识别其中的一些缺陷，我们提出了一些更可靠的性别偏见评估指南，强调了衡量模型性别影响的外在偏差度量的重要性，并发现数据集和度量往往是相互耦合的，这是导致获取可靠结论能力受到阻碍的原因之一。

选择你的视角：性别偏见评估中的缺陷

Choose Your Lenses: Flaws in Gender Bias Evaluation

Multiple metrics have been introduced to measure fairness in various natural
language processing tasks. These metrics can be roughly categorized into two
categories: 1) \emph{extrinsic metrics} for evaluating fairness in downstream
applications and 2) \emph{intrinsic metrics} for estimating fairness in
upstream contextualized language representation models. In this paper, we
conduct an extensive correlation study between intrinsic and extrinsic metrics
across bias notions using 19 contextualized language models. We find that
intrinsic and extrinsic metrics do not necessarily correlate in their original
setting, even when correcting for metric misalignments, noise in evaluation
datasets, and confounding factors such as experiment configuration for
extrinsic metrics. %al

本文对 19 个语境化语言模型进行了广泛的相关性研究，并发现即使在校正度量不准确、评估数据集中有噪声以及实验配置方式等混淆因素的情况下，内在度量和外在度量不一定在其原始设置中相关。