Studies have shown that large pretrained language models exhibit biases against social groups based on race, gender etc, which they inherit from the datasets they are trained on. Various researchers have proposed mathematical tools for quantifying and identifying these biases. There have been methods proposed to mitigate such biases. In this paper, we present a comprehensive quantitative evaluation of different kinds of biases such as race, gender, ethnicity, age etc. exhibited by popular pretrained language models such as BERT, GPT-2 etc. and also present a toolkit that provides plug-and-play interfaces to connect mathematical tools to identify biases with large pretrained language models such as BERT, GPT-2 etc. and also present users with the opportunity to test custom models against these metrics. The toolkit also allows users to debias existing and custom models using the debiasing techniques proposed so far. The toolkit is available at https://github.com/HrishikeshVish/Fairpy.

本文全面评估了常用的预训练语言模型（如BERT、GPT-2等）在种族、性别、种族、年龄等方面所表现出的各种偏见，并介绍了一种工具包，提供了插入数学工具程序以识别偏见的接口，并让用户使用这些度量来测试现有的和自定义的模型。此工具还具有消除偏见的功能。

FairPy：一个大型语言模型的社会偏见评估与缓解工具包