Measures of algorithmic fairness are usually discussed in the context of
binary decisions. We extend the approach to continuous scores. So far,
ROC-based measures have mainly been suggested for this purpose. Other existing
methods depend heavily on the distribution of scores, are unsuitable for
ranking tasks, or their effect sizes are not interpretable. Here, we propose a
distributionally invariant version of fairness measures for continuous scores
with a reasonable interpretation based on the Wasserstein distance. Our
measures are easily computable and well suited for quantifying and interpreting
the strength of group disparities as well as for comparing biases across
different models, datasets, or time points. We derive a link between the
different families of existing fairness measures for scores and show that the
proposed distributionally invariant fairness measures outperform ROC-based
fairness measures because they are more explicit and can quantify significant
biases that ROC-based fairness measures miss. Finally, we demonstrate their
effectiveness through experiments on the most commonly used fairness benchmark
datasets.

通过分布不变公平性度量，我们提出了一套对连续计分进行公平性度量的方法，该方法基于 Wasserstein 距离，具有合理的解释性，并且适用于量化和解释群组差异的强度，以及在不同模型、数据集或时间点之间比较偏差。我们还证明了这些度量的有效性，通过对常用公平性基准数据集的实验验证。

连续分数的可解释分布不变公平度量

Interpretable Distribution-Invariant Fairness Measures for Continuous  Scores

Automatic speech recognition systems based on deep learning are mainly
trained under empirical risk minimization (ERM). Since ERM utilizes the
averaged performance on the data samples regardless of a group such as healthy
or dysarthric speakers, ASR systems are unaware of the performance disparities
across the groups. This results in biased ASR systems whose performance
differences among groups are severe. In this study, we aim to improve the ASR
system in terms of group robustness for dysarthric speakers. To achieve our
goal, we present a novel approach, sample reweighting with sample affinity test
(Re-SAT). Re-SAT systematically measures the debiasing helpfulness of the given
data sample and then mitigates the bias by debiasing helpfulness-based sample
reweighting. Experimental results demonstrate that Re-SAT contributes to
improved ASR performance on dysarthric speech without performance degradation
on healthy speech.

本研究旨在提高针对口吃患者的自动语音识别系统的群体鲁棒性。采用 Re-SAT 方法对数据样本进行去偏倚加权处理，得到的实验结果显示，该方法能够在口吃患者的语音识别上提高系统性能而不会对健康语音的性能造成恶化。

通过样本亲和测试进行样本重新加权，为发音障碍人群提供去偏差的自动语音识别

Debiased Automatic Speech Recognition for Dysarthric Speech via Sample  Reweighting with Sample Affinity Test

We present a benchmark suite of four datasets for evaluating the fairness of
pre-trained language models and the techniques used to fine-tune them for
downstream tasks. Our benchmarks cover four jurisdictions (European Council,
USA, Switzerland, and China), five languages (English, German, French, Italian
and Chinese) and fairness across five attributes (gender, age, region,
language, and legal area). In our experiments, we evaluate pre-trained language
models using several group-robust fine-tuning techniques and show that
performance group disparities are vibrant in many cases, while none of these
techniques guarantee fairness, nor consistently mitigate group disparities.
Furthermore, we provide a quantitative and qualitative analysis of our results,
highlighting open challenges in the development of robustness methods in legal
NLP.

本文提出了一个数据基准，用于评估预训练的语言模型在下游任务中的公平性，包括四个司法管辖区（欧洲理事会、美国、瑞士和中国）、五种语言（英语、德语、法语、意大利语和汉语）以及五个属性的公平性（性别、年龄、地区、语言和法律领域），并通过几种群组鲁棒性的微调技术评估预训练的语言模型，发现在许多情况下，绩效群体差异非常明显，而这些技术均不能保证公平性，并且不能一致地减少群体差异，此外，对结果进行了定量和定性分析，并强调法律 NLP 鲁棒性方法发展中存在的挑战。