Multilingual language models were shown to allow for nontrivial transfer
across scripts and languages. In this work, we study the structure of the
internal representations that enable this transfer. We focus on the
representation of gender distinctions as a practical case study, and examine
the extent to which the gender concept is encoded in shared subspaces across
different languages. Our analysis shows that gender representations consist of
several prominent components that are shared across languages, alongside
language-specific components. The existence of language-independent and
language-specific components provides an explanation for an intriguing
empirical observation we make: while gender classification transfers well
across languages, interventions for gender removal, trained on a single
language, do not transfer easily to others.

本文旨在研究多语言语言模型内部表征结构，以性别划分表示为实用案例，分析不同语言之间共享的子空间中性别概念的编码程度，发现共享的显著成分与语言特定成分并存，这解释了一个现象：尽管性别分类在不同语言之间转移表现良好，但在单一语言上训练的性别去除干预并不容易转移到其他语言。

多语言模型中的性别表征分析

Analyzing Gender Representation in Multilingual Models

Massively multilingual models for neural machine translation (NMT) are
theoretically attractive, but often underperform bilingual models and deliver
poor zero-shot translations. In this paper, we explore ways to improve them. We
argue that multilingual NMT requires stronger modeling capacity to support
language pairs with varying typological characteristics, and overcome this
bottleneck via language-specific components and deepening NMT architectures. We
identify the off-target translation issue (i.e. translating into a wrong target
language) as the major source of the inferior zero-shot performance, and
propose random online backtranslation to enforce the translation of unseen
training language pairs. Experiments on OPUS-100 (a novel multilingual dataset
with 100 languages) show that our approach substantially narrows the
performance gap with bilingual models in both one-to-many and many-to-many
settings, and improves zero-shot performance by ~10 BLEU, approaching
conventional pivot-based methods.

本文探讨了如何改善大规模多语言神经机器翻译模型的性能，并提出了加强模型能力、引入语言特定组件和加深神经机器翻译结构以支持具有不同类型学特征的语言对，同时通过随机在线回译来解决离线训练中未出现的语言对翻译问题。实验结果表明，本方法在一对多和多对多设置中缩小了双语模型的性能差距，并将零 - shot 表现提高约 10 BLEU，接近传统的基于中间语言的方法。