Although large pre-trained language models have achieved great success in
many NLP tasks, it has been shown that they reflect human biases from their
pre-training corpora. This bias may lead to undesirable outcomes when these
models are applied in real-world settings. In this paper, we investigate the
bias present in monolingual BERT models across a diverse set of languages
(English, Greek, and Persian). While recent research has mostly focused on
gender-related biases, we analyze religious and ethnic biases as well and
propose a template-based method to measure any kind of bias, based on sentence
pseudo-likelihood, that can handle morphologically complex languages with
gender-based adjective declensions. We analyze each monolingual model via this
method and visualize cultural similarities and differences across different
dimensions of bias. Ultimately, we conclude that current methods of probing for
bias are highly language-dependent, necessitating cultural insights regarding
the unique ways bias is expressed in each language and culture (e.g. through
coded language, synecdoche, and other similar linguistic concepts). We also
hypothesize that higher measured social biases in the non-English BERT models
correlate with user-generated content in their training.

本研究探讨了预训练语言模型在不同语言中所存在的偏见，提出了一种基于句子假设度量的模板方法来检测偏见，通过该方法分析了每个单语模型。研究发现，检测偏见的方法高度依赖于语言和文化，需要对每种语言和文化的独特表达方式进行深入了解。而与此相关的，我们假设非英语 BERT 模型中更高的社会偏见指数与其训练的用户生成内容有关。