In most recent studies, gender bias in document ranking is evaluated with the
NFaiRR metric, which measures bias in a ranked list based on an aggregation
over the unbiasedness scores of each ranked document. This perspective in
measuring the bias of a ranked list has a key limitation: individual documents
of a ranked list might be biased while the ranked list as a whole balances the
groups' representations. To address this issue, we propose a novel metric
called TExFAIR (term exposure-based fairness), which is based on two new
extensions to a generic fairness evaluation framework, attention-weighted
ranking fairness (AWRF). TExFAIR assesses fairness based on the term-based
representation of groups in a ranked list: (i) an explicit definition of
associating documents to groups based on probabilistic term-level associations,
and (ii) a rank-biased discounting factor (RBDF) for counting
non-representative documents towards the measurement of the fairness of a
ranked list. We assess TExFAIR on the task of measuring gender bias in passage
ranking, and study the relationship between TExFAIR and NFaiRR. Our experiments
show that there is no strong correlation between TExFAIR and NFaiRR, which
indicates that TExFAIR measures a different dimension of fairness than NFaiRR.
With TExFAIR, we extend the AWRF framework to allow for the evaluation of
fairness in settings with term-based representations of groups in documents in
a ranked list.

通过引入 TExFAIR 度量方法，该研究提供了一种新的评估框架，用于衡量文件排名中的性别偏见和公平性，通过基于术语的组表示来评估排名列表中性别偏见的公平性，与现有的评估方法 NFaiRR 有所不同。