Deep and large pre-trained language models are the state-of-the-art for various natural language processing tasks. However, the huge size of these models could be a deterrent to use them in practice. Some recent and concurrent works use knowledge distillation to compress these huge models into shallow ones. In this work we study knowledge distillation with a focus on multi-lingual Named Entity Recognition (NER). In particular, we study several distillation strategies and propose a stage-wise optimization scheme leveraging teacher internal representations that is agnostic of teacher architecture and show that it outperforms strategies employed in prior works. Additionally, we investigate the role of several factors like the amount of unlabeled data, annotation resources, model architecture and inference latency to name a few. We show that our approach leads to massive compression of MBERT-like teacher models by upto 35x in terms of parameters and 51x in terms of latency for batch inference while retaining 95% of its F1-score for NER over 41 languages.

本研究聚焦于多语言实体识别，探究知识蒸馏压缩预训练语言模型的多种策略，通过利用教师模型内部表示的分阶段优化方案，成功将MBERT模型压缩了35倍参数，51倍批量推理的延迟，同时保持在41种语言中的95％的F1分数。

XtremeDistil：大规模多语言模型的多阶段蒸馏