It has been shown that multilingual BERT (mBERT) yields high quality
multilingual representations and enables effective zero-shot transfer. This is
surprising given that mBERT does not use any crosslingual signal during
training. While recent literature has studied this phenomenon, the reasons for
the multilinguality are still somewhat obscure. We aim to identify
architectural properties of BERT and linguistic properties of languages that
are necessary for BERT to become multilingual. To allow for fast
experimentation we propose an efficient setup with small BERT models trained on
a mix of synthetic and natural data. Overall, we identify four architectural
and two linguistic elements that influence multilinguality. Based on our
insights, we experiment with a multilingual pretraining setup that modifies the
masking strategy using VecMap, i.e., unsupervised embedding alignment.
Experiments on XNLI with three languages indicate that our findings transfer
from our small setup to larger scale settings.

该研究通过实现小型 BERT 模型的混合合成数据和自然数据训练，试图从语言学和结构特征两个方面，探究多语 BERT 能实现无监督跨语言转移的原因。其结果表明，在 lexical、syntactic 以及阅读理解方面，mBERT 已实现了高质量的多语言表征和跨语言转移功能。