Masked Language Models (MLMs) pre-trained by predicting masked tokens on
large corpora have been used successfully in natural language processing tasks
for a variety of languages. Unfortunately, it was reported that MLMs also learn
discriminative biases regarding attributes such as gender and race. Because
most studies have focused on MLMs in English, the bias of MLMs in other
languages has rarely been investigated. Manual annotation of evaluation data
for languages other than English has been challenging due to the cost and
difficulty in recruiting annotators. Moreover, the existing bias evaluation
methods require the stereotypical sentence pairs consisting of the same context
with attribute words (e.g. He/She is a nurse). We propose Multilingual Bias
Evaluation (MBE) score, to evaluate bias in various languages using only
English attribute word lists and parallel corpora between the target language
and English without requiring manually annotated data. We evaluated MLMs in
eight languages using the MBE and confirmed that gender-related biases are
encoded in MLMs for all those languages. We manually created datasets for
gender bias in Japanese and Russian to evaluate the validity of the MBE. The
results show that the bias scores reported by the MBE significantly correlates
with that computed from the above manually created datasets and the existing
English datasets for gender bias.

通过提出多语种偏差评估模型（MBE）和手动创建的数据集对八种语言的语言模型进行评估，证实了所有这些语言中均存在针对性别相关的偏差，同时在日语和俄语中的手动创建的数据集与 MBE 评分存在显著相关性。