Multilingual sequence labeling is a task of predicting label sequences using
a single unified model for multiple languages. Compared with relying on
multiple monolingual models, using a multilingual model has the benefit of a
smaller model size, easier in online serving, and generalizability to
low-resource languages. However, current multilingual models still underperform
individual monolingual models significantly due to model capacity limitations.
In this paper, we propose to reduce the gap between monolingual models and the
unified multilingual model by distilling the structural knowledge of several
monolingual models (teachers) to the unified multilingual model (student). We
propose two novel KD methods based on structure-level information: (1)
approximately minimizes the distance between the student's and the teachers'
structure level probability distributions, (2) aggregates the structure-level
knowledge to local distributions and minimizes the distance between two local
probability distributions. Our experiments on 4 multilingual tasks with 25
datasets show that our approaches outperform several strong baselines and have
stronger zero-shot generalizability than both the baseline model and teacher
models.

该研究提出使用知识蒸馏减少多语言模型与单语言模型之间性能差距，实验结果显示我们的方法优于多个基准模型并具有更强的零次通用性。

多语言序列标注的结构级知识蒸馏

Structure-Level Knowledge Distillation For Multilingual Sequence  Labeling

We propose a practical scheme to train a single multilingual sequence
labeling model that yields state of the art results and is small and fast
enough to run on a single CPU. Starting from a public multilingual BERT
checkpoint, our final model is 6x smaller and 27x faster, and has higher
accuracy than a state-of-the-art multilingual baseline. We show that our model
especially outperforms on low-resource languages, and works on codemixed input
text without being explicitly trained on codemixed examples. We showcase the
effectiveness of our method by reporting on part-of-speech tagging and
morphological prediction on 70 treebanks and 48 languages.

我们提出了一种实用的方案来训练一个单一的多语言序列标注模型，这个模型在单个 CPU 上运行，能够给出最先进的结果，足够小和快速。从一个公共的多语言 BERT 检查点开始，我们的最终模型比最先进的多语言基线模型更加准确，并且速度更快，6 倍更小。我们证明了我们的模型特别是在低资源语言上表现出色，并且可以在混合输入文本上工作，而不需要对混合示例进行明确的训练。我们通过 70 棵树库和 48 种语言的词汇标注和形态预测来展示我们方法的有效性。