Transfer learning based on pretraining language models on a large amount of
raw data has become a new norm to reach state-of-the-art performance in NLP.
Still, it remains unclear how this approach should be applied for unseen
languages that are not covered by any available large-scale multilingual
language model and for which only a small amount of raw data is generally
available. In this work, by comparing multilingual and monolingual models, we
show that such models behave in multiple ways on unseen languages. Some
languages greatly benefit from transfer learning and behave similarly to
closely related high resource languages whereas others apparently do not.
Focusing on the latter, we show that this failure to transfer is largely
related to the impact of the script used to write such languages.
Transliterating those languages improves very significantly the ability of
large-scale multilingual language models on downstream tasks.

通过比较多语言和单语言模型，本文表明了在大量的原始数据上使用基于预训练语言模型的迁移学习在处理未见过的语言中存在多种行为，其中一些语言能够从迁移学习中受益，而另一些则似乎不行，我们还发现这种无法传递的失败很大程度上与用于书写这些语言的字符的影响有关，转写这些语言可以极大地提高后续任务的大规模多语言语言模型的能力。