Natural Language Processing (NLP) plays an important role in our daily lives,
particularly due to the enormous progress of Large Language Models (LLM).
However, NLP has many fairness-critical use cases, e.g., as an expert system in
recruitment or as an LLM-based tutor in education. Since NLP is based on human
language, potentially harmful biases can diffuse into NLP systems and produce
unfair results, discriminate against minorities or generate legal issues.
Hence, it is important to develop a fairness certification for NLP approaches.
We follow a qualitative research approach towards a fairness certification for
NLP. In particular, we have reviewed a large body of literature on algorithmic
fairness, and we have conducted semi-structured expert interviews with a wide
range of experts from that area. We have systematically devised six fairness
criteria for NLP, which can be further refined into 18 sub-categories. Our
criteria offer a foundation for operationalizing and testing processes to
certify fairness, both from the perspective of the auditor and the audited
organization.

发展公平认证以确保自然语言处理系统的算法公正性和对少数群体的非歧视性。

自然语言处理和大型语言模型的公平认证

Fairness Certification for Natural Language Processing and Large  Language Models

We propose a method for certifying the fairness of the classification result
of a widely used supervised learning algorithm, the k-nearest neighbors (KNN),
under the assumption that the training data may have historical bias caused by
systematic mislabeling of samples from a protected minority group. To the best
of our knowledge, this is the first certification method for KNN based on three
variants of the fairness definition: individual fairness, $\epsilon$-fairness,
and label-flipping fairness. We first define the fairness certification problem
for KNN and then propose sound approximations of the complex arithmetic
computations used in the state-of-the-art KNN algorithm. This is meant to lift
the computation results from the concrete domain to an abstract domain, to
reduce the computational cost. We show effectiveness of this abstract
interpretation based technique through experimental evaluation on six datasets
widely used in the fairness research literature. We also show that the method
is accurate enough to obtain fairness certifications for a large number of test
inputs, despite the presence of historical bias in the datasets.

我们提出了一种方法，用于证明广泛使用的监督学习算法 k 最近邻（KNN）的分类结果的公平性，假设训练数据可能存在由于对受保护的少数群体的样本进行系统性错误标记而导致的历史偏差。首次基于个体公平性、ε- 公平性和标签翻转公平性三个公平定义的 KNN 公平性认证方法。我们首先定义了 KNN 的公平性认证问题，然后提出了用于复杂算术计算的声音近似方法，该方法用于最先进的 KNN 算法中，旨在将计算结果从具体域提升到抽象域以减少计算成本。通过对在公平性研究文献中广泛使用的六个数据集进行实验评估，我们展示了这种基于抽象解释的技术的有效性。我们还展示了尽管数据集中存在历史偏差，但该方法足够准确以获取大量测试输入的公平性认证。

证明 KNN 在数据集偏见存在下的公正性

Certifying the Fairness of KNN in the Presence of Dataset Bias

This paper proposes a novel framework for certifying the fairness of
predictive models trained on biased data. It draws from query answering for
incomplete and inconsistent databases to formulate the problem of consistent
range approximation (CRA) of fairness queries for a predictive model on a
target population. The framework employs background knowledge of the data
collection process and biased data, working with or without limited statistics
about the target population, to compute a range of answers for fairness
queries. Using CRA, the framework builds predictive models that are certifiably
fair on the target population, regardless of the availability of external data
during training. The framework's efficacy is demonstrated through evaluations
on real data, showing substantial improvement over existing state-of-the-art
methods.

提出了一种新的框架，用于验证在偏差数据上训练的预测模型的公平性，其利用了数据库不完整和不一致的查询方法来解决在目标人群上进行公平性查询的一致性范围逼近 (CRA) 问题。在有限的目标人群统计数据或者没有这些数据的情况下，该框架利用数据收集过程和偏差数据的背景知识计算公平性查询的答案范围，并构建可验证地公平的预测模型。实验结果证明了该框架的有效性，相比现有最先进的方法有大幅度提高。