Multiple metrics have been introduced to measure fairness in various natural
language processing tasks. These metrics can be roughly categorized into two
categories: 1) \emph{extrinsic metrics} for evaluating fairness in downstream
applications and 2) \emph{intrinsic metrics} for estimating fairness in
upstream contextualized language representation models. In this paper, we
conduct an extensive correlation study between intrinsic and extrinsic metrics
across bias notions using 19 contextualized language models. We find that
intrinsic and extrinsic metrics do not necessarily correlate in their original
setting, even when correcting for metric misalignments, noise in evaluation
datasets, and confounding factors such as experiment configuration for
extrinsic metrics. %al

本文对 19 个语境化语言模型进行了广泛的相关性研究，并发现即使在校正度量不准确、评估数据集中有噪声以及实验配置方式等混淆因素的情况下，内在度量和外在度量不一定在其原始设置中相关。

关于上下文语言表示的内在公正性和外在公正性评估指标

On the Intrinsic and Extrinsic Fairness Evaluation Metrics for Contextualized Language Representations

Although there is an emerging trend towards generating embeddings for
primarily unstructured data and, recently, for structured data, no systematic
suite for measuring the quality of embeddings has been proposed yet. This
deficiency is further sensed with respect to embeddings generated for
structured data because there are no concrete evaluation metrics measuring the
quality of the encoded structure as well as semantic patterns in the embedding
space. In this paper, we introduce a framework containing three distinct tasks
concerned with the individual aspects of ontological concepts: (i) the
categorization aspect, (ii) the hierarchical aspect, and (iii) the relational
aspect. Then, in the scope of each task, a number of intrinsic metrics are
proposed for evaluating the quality of the embeddings. Furthermore, w.r.t. this
framework, multiple experimental studies were run to compare the quality of the
available embedding models. Employing this framework in future research can
reduce misjudgment and provide greater insight about quality comparisons of
embeddings for ontological concepts. We positioned our sampled data and code at
this https URL under GNU General Public License v3.0.

提出一个包含三个不同任务的框架，关注本体概念的不同方面，并为框架中的每个任务提出了多个内部度量标准，以评估嵌入质量，进而比较可用的嵌入模型的质量，以在将来的研究中使用该框架来减少错误判断并提供更深入的了解。