Gender bias is largely recognized as a problematic phenomenon affecting
language technologies, with recent studies underscoring that it might surface
differently across languages. However, most of current evaluation practices
adopt a word-level focus on a narrow set of occupational nouns under synthetic
conditions. Such protocols overlook key features of grammatical gender
languages, which are characterized by morphosyntactic chains of gender
agreement, marked on a variety of lexical items and parts-of-speech (POS). To
overcome this limitation, we enrich the natural, gender-sensitive MuST-SHE
corpus (Bentivogli et al., 2020) with two new linguistic annotation layers (POS
and agreement chains), and explore to what extent different lexical categories
and agreement phenomena are impacted by gender skews. Focusing on speech
translation, we conduct a multifaceted evaluation on three language directions
(English-French/Italian/Spanish), with models trained on varying amounts of
data and different word segmentation techniques. By shedding light on model
behaviours, gender bias, and its detection at several levels of granularity,
our findings emphasize the value of dedicated analyses beyond aggregated
overall results.

通过对自然、性别敏感的 MuST-SHE 语料库进行 POS 和一致性链的语言学注释，在英法 / 意大利语 / 西班牙语三种语言方向上针对不同数量的数据和不同的词语分割技术进行多方面评估，揭示了性别偏见的模型行为，及其在多个细粒度层次上的检测价值。