We study the problem of multilingual masked language modeling, i.e. the
training of a single model on concatenated text from multiple languages, and
present a detailed study of several factors that influence why these models are
so effective for cross-lingual transfer. We show, contrary to what was
previously hypothesized, that transfer is possible even when there is no shared
vocabulary across the monolingual corpora and also when the text comes from
very different domains. The only requirement is that there are some shared
parameters in the top layers of the multi-lingual encoder. To better understand
this result, we also show that representations from independently trained
models in different languages can be aligned post-hoc quite effectively,
strongly suggesting that, much like for non-contextual word embeddings, there
are universal latent symmetries in the learned embedding spaces. For
multilingual masked language modeling, these symmetries seem to be
automatically discovered and aligned during the joint training process.

通过研究多语言掩码语言建模问题，我们阐述了若干因素，即为何这些模型在跨语言转移方面如此有效，并证明了即使单语料库之间没有共享词汇，也可以进行转移，只需在多语言编码器顶层存在一些共享参数。同时，我们展示了来自不同语言、独立训练模型的表示可以很好地进行后期对齐，这些模型似乎自动发现和对齐了学习嵌入空间中的普遍潜在对称性。对于多语言掩码语言建模，这些对称性似乎是在联合训练过程中自动发现和对齐的。