Multilingual pre-trained language models (MPLMs) not only can handle tasks in
different languages but also exhibit surprising zero-shot cross-lingual
transferability. However, MPLMs usually are not able to achieve comparable
supervised performance on rich-resource languages compared to the
state-of-the-art monolingual pre-trained models. In this paper, we aim to
improve the multilingual model's supervised and zero-shot performance
simultaneously only with the resources from supervised languages. Our approach
is based on transferring knowledge from high-performance monolingual models
with a teacher-student framework. We let the multilingual model learn from
multiple monolingual models simultaneously. To exploit the model's
cross-lingual transferability, we propose MBLM (multi-branch multilingual
language model), a model built on the MPLMs with multiple language branches.
Each branch is a stack of transformers. MBLM is trained with the
zero-shot-aware training strategy that encourages the model to learn from the
mixture of zero-shot representations from all the branches. The results on two
cross-lingual classification tasks show that, with only the task's supervised
data used, our method improves both the supervised and zero-shot performance of
MPLMs.

利用教师 - 学生框架从高性能的单语言模型中转移知识，构建了一个基于 MPLMs 的多语言分支模型（MBLM），并使用零射击感知的训练策略令模型从所有分支的零射击表示中学习，我们的方法仅使用任务的监督数据，提高了 MPLMs 的监督性能和零射击性能。