Pre-trained multilingual language models play an important role in
cross-lingual natural language understanding tasks. However, existing methods
did not focus on learning the semantic structure of representation, and thus
could not optimize their performance. In this paper, we propose Multi-level
Multilingual Knowledge Distillation (MMKD), a novel method for improving
multilingual language models. Specifically, we employ a teacher-student
framework to adopt rich semantic representation knowledge in English BERT. We
propose token-, word-, sentence-, and structure-level alignment objectives to
encourage multiple levels of consistency between source-target pairs and
correlation similarity between teacher and student models. We conduct
experiments on cross-lingual evaluation benchmarks including XNLI, PAWS-X, and
XQuAD. Experimental results show that MMKD outperforms other baseline models of
similar size on XNLI and XQuAD and obtains comparable performance on PAWS-X.
Especially, MMKD obtains significant performance gains on low-resource
languages.

本论文提出了一种新的多层次多语种知识蒸馏方法（MMKD），采用英语 BERT 中的丰富语义表征知识和师生框架来鼓励源 - 目标对之间的多个层次一致性和教师和学生模型之间的相关相似性，以提高预先训练的多语种语言模型的性能。在横跨语言的评价任务中进行了实验证明，相对其他基线模型，MMKD 在 XNLI 和 XQuAD 上表现更好，在 PAWS-X 上表现相当，并且在低资源语言上获得了显著的性能提升。