Language models (LMs) have become pivotal in the realm of technological
advancements. While their capabilities are vast and transformative, they often
include societal biases encoded in the human-produced datasets used for their
training. This research delves into the inherent biases present in masked
language models (MLMs), with a specific focus on gender biases. This study
evaluated six prominent models: BERT, RoBERTa, DistilBERT, BERT-multilingual,
XLM-RoBERTa, and DistilBERT-multilingual. The methodology employed a novel
dataset, bifurcated into two subsets: one containing prompts that encouraged
models to generate subject pronouns in English, and the other requiring models
to return the probabilities of verbs, adverbs, and adjectives linked to the
prompts' gender pronouns. The analysis reveals stereotypical gender alignment
of all models, with multilingual variants showing comparatively reduced biases.

该研究通过评估六个主要的语言模型（BERT，RoBERTa，DistilBERT，BERT-multilingual，XLM-RoBERTa 和 DistilBERT-multilingual）并采用包含鼓励模型生成英语主题代词和要求模型返回与性别代词相关的动词、副词和形容词概率的提示来调查遮蔽语言模型中固有的偏见，尤其是性别偏见。分析结果显示所有模型存在性别刻板印象，而多语言变体的偏见相对较小。

UnMASKed: 通过语言学知识的职业市场提示量化遮蔽语言模型中的性别偏见

UnMASKed: Quantifying Gender Biases in Masked Language Models through  Linguistically Informed Job Market Prompts

The T5 model and its unified text-to-text paradigm contributed in advancing
the state-of-the-art for many natural language processing tasks. While some
multilingual variants of the T5 model have recently been introduced, their
performances were found to provide suboptimal performances for languages other
than English if compared to monolingual variants. We are motivated by these
findings to introduce IT5, the first family of encoder-decoder transformer
models pretrained specifically on Italian. We perform a thorough cleaning of a
web-crawled Italian corpus including more than 40 billion words and use it to
pretrain three IT5 models of different sizes. The performance of IT5 models and
their multilingual counterparts is then evaluated on a broad range of natural
language understanding and generation benchmarks for Italian. We find the
monolingual IT5 models to provide the best scale-to-performance ratio across
tested models, consistently outperforming their multilingual counterparts and
setting a new state-of-the-art for most Italian conditional language generation
tasks.

引入了 IT5 模型，这是首个在意大利语上进行预训练的编码器 - 解码器 Transformer 模型家族，研究表明，它在大多数条件语言生成任务中表现出比其多语种对应物更好的性能。