We present a benchmark suite of four datasets for evaluating the fairness of
pre-trained language models and the techniques used to fine-tune them for
downstream tasks. Our benchmarks cover four jurisdictions (European Council,
USA, Switzerland, and China), five languages (English, German, French, Italian
and Chinese) and fairness across five attributes (gender, age, region,
language, and legal area). In our experiments, we evaluate pre-trained language
models using several group-robust fine-tuning techniques and show that
performance group disparities are vibrant in many cases, while none of these
techniques guarantee fairness, nor consistently mitigate group disparities.
Furthermore, we provide a quantitative and qualitative analysis of our results,
highlighting open challenges in the development of robustness methods in legal
NLP.

本文提出了一个数据基准，用于评估预训练的语言模型在下游任务中的公平性，包括四个司法管辖区（欧洲理事会、美国、瑞士和中国）、五种语言（英语、德语、法语、意大利语和汉语）以及五个属性的公平性（性别、年龄、地区、语言和法律领域），并通过几种群组鲁棒性的微调技术评估预训练的语言模型，发现在许多情况下，绩效群体差异非常明显，而这些技术均不能保证公平性，并且不能一致地减少群体差异，此外，对结果进行了定量和定性分析，并强调法律 NLP 鲁棒性方法发展中存在的挑战。