Knowledge distillation is of key importance to launching multilingual pre-trained language models for real applications. To support cost-effective language inference in multilingual settings, we propose AMTSS, an adaptive multi-teacher single-student distillation framework, which allows distilling knowledge from multiple teachers to a single student. We first introduce an adaptive learning strategy and teacher importance weight, which enables a student to effectively learn from max-margin teachers and easily adapt to new languages. Moreover, we present a shared student encoder with different projection layers in support of multiple languages, which contributes to largely reducing development and machine cost. Experimental results show that AMTSS gains competitive results on the public XNLI dataset and the realistic industrial dataset AliExpress (AE) in the E-commerce scenario.

提出AMTSS，一种自适应的多教师单学生蒸馏框架，它允许从多个教师向单个学生蒸馏知识，对于开发和机器成本的大幅降低做出了贡献。在公共XNLI数据集和电子商务情景下的实际工业数据集AliExpress（AE）上获得了有竞争力的结果。

AMTSS: 自适应多教师单学生知识蒸馏框架用于多语言语言推断