Recently, multilingual BERT works remarkably well on cross-lingual transfer
tasks, superior to static non-contextualized word embeddings. In this work, we
provide an in-depth experimental study to supplement the existing literature of
cross-lingual ability. We compare the cross-lingual ability of
non-contextualized and contextualized representation model with the same data.
We found that datasize and context window size are crucial factors to the
transferability. We also observe the language-specific information in
multilingual BERT. By manipulating the latent representations, we can control
the output languages of multilingual BERT, and achieve unsupervised token
translation. We further show that based on the observation, there is a
computationally cheap but effective approach to improve the cross-lingual
ability of multilingual BERT.

本研究针对跨语言能力对多语言 BERT 进行了深入实验研究，发现数据大小和上下文窗口大小是跨语言能力的关键因素，探讨了多语言 BERT 中的语言特定信息，通过操作潜在表示控制多语言 BERT 的语言输出，实现了无监督令牌翻译，并提出了一种计算成本低但有效提高多语言 BERT 跨语言能力的方法。