Although large pre-trained language models have achieved great success in
many NLP tasks, it has been shown that they reflect human biases from their
pre-training corpora. This bias may lead to undesirable outcomes when these
models are applied in real-world settings. In this paper, we investigate the
bias present in monolingual BERT models across a diverse set of languages
(English, Greek, and Persian). While recent research has mostly focused on
gender-related biases, we analyze religious and ethnic biases as well and
propose a template-based method to measure any kind of bias, based on sentence
pseudo-likelihood, that can handle morphologically complex languages with
gender-based adjective declensions. We analyze each monolingual model via this
method and visualize cultural similarities and differences across different
dimensions of bias. Ultimately, we conclude that current methods of probing for
bias are highly language-dependent, necessitating cultural insights regarding
the unique ways bias is expressed in each language and culture (e.g. through
coded language, synecdoche, and other similar linguistic concepts). We also
hypothesize that higher measured social biases in the non-English BERT models
correlate with user-generated content in their training.

本研究探讨了预训练语言模型在不同语言中所存在的偏见，提出了一种基于句子假设度量的模板方法来检测偏见，通过该方法分析了每个单语模型。研究发现，检测偏见的方法高度依赖于语言和文化，需要对每种语言和文化的独特表达方式进行深入了解。而与此相关的，我们假设非英语 BERT 模型中更高的社会偏见指数与其训练的用户生成内容有关。

分析多种语言 BERT 变体中存在的社会偏见

An Analysis of Social Biases Present in BERT Variants Across Multiple Languages

The study of the applicability of the BERTScore metric was conducted to
translation quality assessment at the sentence level for English -> Russian
direction. Experiments were performed with a pre-trained Multilingual BERT as
well as with a pair of Monolingual BERT models. To align monolingual
embeddings, an orthogonal transformation based on anchor tokens was used. It
was demonstrated that such transformation helps to prevent mismatching issue
and shown that this approach gives better results than using embeddings of the
Multilingual model. To improve the token matching process it is proposed to
combine all incomplete WorkPiece tokens into meaningful words and use simple
averaging of corresponding vectors and to calculate BERTScore based on anchor
tokens only. Such modifications allowed us to achieve a better correlation of
the model predictions with human judgments. In addition to evaluating machine
translation, several versions of human translation were evaluated as well, the
problems of this approach were listed.

该研究旨在利用 BERTScore 度量英俄翻译中句子级别的翻译质量，并通过利用单语 BERT 模型和基于锚定标记的正交变换对齐单语言嵌入来改进模型性能，结果表明改进的方法可以提高模型预测与人类判断的相关性。