Discriminatory social biases, including gender biases, have been found in
Pre-trained Language Models (PLMs). In Natural Language Inference (NLI), recent
bias evaluation methods have observed biased inferences from the outputs of a
particular label such as neutral or entailment. However, since different biased
inferences can be associated with different output labels, it is inaccurate for
a method to rely on one label. In this work, we propose an evaluation method
that considers all labels in the NLI task. We create evaluation data and assign
them into groups based on their expected biased output labels. Then, we define
a bias measure based on the corresponding label output of each data group. In
the experiment, we propose a meta-evaluation method for NLI bias measures, and
then use it to confirm that our measure can evaluate bias more accurately than
the baseline. Moreover, we show that our evaluation method is applicable to
multiple languages by conducting the meta-evaluation on PLMs in three different
languages: English, Japanese, and Chinese. Finally, we evaluate PLMs of each
language to confirm their bias tendency. To our knowledge, we are the first to
build evaluation datasets and measure the bias of PLMs from the NLI task in
Japanese and Chinese.

本文提出了一种考虑自然语言推理任务中所有标签的评估方法来评估预训练语言模型中的偏见，并通过对多种语言的模型进行元评估，证明该方法能够更准确地评估偏见。此外，本文还是首次在日语和中文的自然语言推理任务中构建评估数据集并测量预训练语言模型的偏见倾向。