Multilingual pre-trained models have demonstrated their effectiveness in many
multilingual NLP tasks and enabled zero-shot or few-shot transfer from
high-resource languages to low resource ones. However, due to significant
typological differences and contradictions between some languages, such models
usually perform poorly on many languages and cross-lingual settings, which
shows the difficulty of learning a single model to handle massive diverse
languages well at the same time. To alleviate this issue, we present a new
multilingual pre-training pipeline. We propose to generate language
representation from multilingual pre-trained models and conduct linguistic
analysis to show that language representation similarity reflect linguistic
similarity from multiple perspectives, including language family, geographical
sprachbund, lexicostatistics and syntax. Then we cluster all the target
languages into multiple groups and name each group as a representation
sprachbund. Thus, languages in the same representation sprachbund are supposed
to boost each other in both pre-training and fine-tuning as they share rich
linguistic similarity. We pre-train one multilingual model for each
representation sprachbund. Experiments are conducted on cross-lingual
benchmarks and significant improvements are achieved compared to strong
baselines.

介绍了一种新的多语种预训练模型方法，基于语言的相似性将目标语言划分为若干组，并对每组构建一个预训练模型，实验中在跨语言基准测试中与强基线模型相比获得了显著的提升。