Many pretrained multilingual models exhibit cross-lingual transfer ability,
which is often attributed to a learned language-neutral representation during
pretraining. However, it remains unclear what factors contribute to the
learning of a language-neutral representation, and whether the learned
language-neutral representation suffices to facilitate cross-lingual transfer.
We propose a synthetic task, Multilingual Othello (mOthello), as a testbed to
delve into these two questions. We find that: (1) models trained with naive
multilingual pretraining fail to learn a language-neutral representation across
all input languages; (2) the introduction of "anchor tokens" (i.e., lexical
items that are identical across languages) helps cross-lingual representation
alignment; and (3) the learning of a language-neutral representation alone is
not sufficient to facilitate cross-lingual transfer. Based on our findings, we
propose a novel approach - multilingual pretraining with unified output space -
that both induces the learning of language-neutral representation and
facilitates cross-lingual transfer.

预训练的多语言模型在交叉语言迁移方面展现出了能力，本研究旨在探讨学习语言中性表示的因素以及该表示是否足以促进交叉语言迁移。通过引入 “锚定标记”，可以帮助跨语言表示对齐，而仅仅学习语言中性表示是不足以促进交叉语言迁移的。基于研究结果，提出一种新的方法 - 统一输出空间的多语言预训练 - 既能够诱导学习语言中性表示，又能促进交叉语言迁移。