In this paper, we propose Stochastic Knowledge Distillation (SKD) to obtain
compact BERT-style language model dubbed SKDBERT. In each iteration, SKD
samples a teacher model from a pre-defined teacher ensemble, which consists of
multiple teacher models with multi-level capacities, to transfer knowledge into
student model in an one-to-one manner. Sampling distribution plays an important
role in SKD. We heuristically present three types of sampling distributions to
assign appropriate probabilities for multi-level teacher models. SKD has two
advantages: 1) it can preserve the diversities of multi-level teacher models
via stochastically sampling single teacher model in each iteration, and 2) it
can also improve the efficacy of knowledge distillation via multi-level teacher
models when large capacity gap exists between the teacher model and the student
model. Experimental results on GLUE benchmark show that SKDBERT reduces the
size of a BERT$_{\rm BASE}$ model by 40% while retaining 99.5% performances of
language understanding and being 100% faster.

本文提出了随机知识蒸馏（SKD）算法用于获得紧凑的 BERT 风格语言模型，同时在多个教师模型的帮助下提高了知识蒸馏的效力，实验结果表明 SKDBERT 在保持 99.5% 语言理解准确度的同时，压缩了 BERT_BASE 模型 40% 的大小且速度提升 100%。