Multilingual pre-trained language models, such as mBERT and XLM-R, have shown
impressive cross-lingual ability. Surprisingly, both of them use multilingual
masked language model (MLM) without any cross-lingual supervision or aligned
data. Despite the encouraging results, we still lack a clear understanding of
why cross-lingual ability could emerge from multilingual MLM. In our work, we
argue that cross-language ability comes from the commonality between languages.
Specifically, we study three language properties: constituent order,
composition and word co-occurrence. First, we create an artificial language by
modifying property in source language. Then we study the contribution of
modified property through the change of cross-language transfer results on
target language. We conduct experiments on six languages and two cross-lingual
NLP tasks (textual entailment, sentence retrieval). Our main conclusion is that
the contribution of constituent order and word co-occurrence is limited, while
the composition is more crucial to the success of cross-linguistic transfer.

研究了多语言预训练语言模型的跨语言能力，得出结论组成是成功的跨语言转移的关键因素。

多语言遮蔽语言模型的跨语言能力：语言结构研究

Cross-Lingual Ability of Multilingual Masked Language Models: A Study of Language Structure

Multilingual pre-trained language models have shown impressive performance on
cross-lingual tasks. It greatly facilitates the applications of natural
language processing on low-resource languages. However, there are still some
languages that the current multilingual models do not perform well on. In this
paper, we propose CINO (Chinese Minority Pre-trained Language Model), a
multilingual pre-trained language model for Chinese minority languages. It
covers Standard Chinese, Yue Chinese, and six other ethnic minority languages.
To evaluate the cross-lingual ability of the multilingual model on ethnic
minority languages, we collect documents from Wikipedia and news websites, and
construct two text classification datasets, WCM (Wiki-Chinese-Minority) and
CMNews (Chinese-Minority-News). We show that CINO notably outperforms the
baselines on various classification tasks. The CINO model and the datasets are
publicly available at this http URL.

本文提出了 CINO 一种为中国少数民族语言设计的多语言预训练语言模型，该模型表现出良好的跨语言分类性能。作者构建了两个分类数据集 WCM 和 CMNews 以评估跨语言能力。

CINO: 一个中文少数民族预训练语言模型

CINO: A Chinese Minority Pre-trained Language Model

Recent work has exhibited the surprising cross-lingual abilities of
multilingual BERT (M-BERT) -- surprising since it is trained without any
cross-lingual objective and with no aligned data. In this work, we provide a
comprehensive study of the contribution of different components in M-BERT to
its cross-lingual ability. We study the impact of linguistic properties of the
languages, the architecture of the model, and the learning objectives. The
experimental study is done in the context of three typologically different
languages -- Spanish, Hindi, and Russian -- and using two conceptually
different NLP tasks, textual entailment and named entity recognition. Among our
key conclusions is the fact that the lexical overlap between languages plays a
negligible role in the cross-lingual success, while the depth of the network is
an integral part of it. All our models and implementations can be found on our
project page: this http URL .

通过在不同语言和不同自定义 NLP 任务上的实验研究，本文详细研究了 M-BERT 中不同组建在其跨语言能力中的贡献，发现词汇重叠在跨语言成功中起到微不足道的作用，而网络深度则是成功的一个重要因素。