Multilingual pre-trained language models, such as mBERT and XLM-R, have shown
impressive cross-lingual ability. Surprisingly, both of them use multilingual
masked language model (MLM) without any cross-lingual supervision or aligned
data. Despite the encouraging results, we still lack a clear understanding of
why cross-lingual ability could emerge from multilingual MLM. In our work, we
argue that cross-language ability comes from the commonality between languages.
Specifically, we study three language properties: constituent order,
composition and word co-occurrence. First, we create an artificial language by
modifying property in source language. Then we study the contribution of
modified property through the change of cross-language transfer results on
target language. We conduct experiments on six languages and two cross-lingual
NLP tasks (textual entailment, sentence retrieval). Our main conclusion is that
the contribution of constituent order and word co-occurrence is limited, while
the composition is more crucial to the success of cross-linguistic transfer.

研究了多语言预训练语言模型的跨语言能力，得出结论组成是成功的跨语言转移的关键因素。