Language model pre-training, such as BERT, has significantly improved the
performances of many natural language processing tasks. However, pre-trained
language models are usually computationally expensive, so it is difficult to
efficiently execute them on resource-restricted devices. To accelerate
inference and reduce model size while maintaining accuracy, we first propose a
novel Transformer distillation method that is specially designed for knowledge
distillation (KD) of the Transformer-based models. By leveraging this new KD
method, the plenty of knowledge encoded in a large teacher BERT can be
effectively transferred to a small student Tiny-BERT. Then, we introduce a new
two-stage learning framework for TinyBERT, which performs Transformer
distillation at both the pretraining and task-specific learning stages. This
framework ensures that TinyBERT can capture he general-domain as well as the
task-specific knowledge in BERT.
TinyBERT with 4 layers is empirically effective and achieves more than 96.8%
the performance of its teacher BERTBASE on GLUE benchmark, while being 7.5x
smaller and 9.4x faster on inference. TinyBERT with 4 layers is also
significantly better than 4-layer state-of-the-art baselines on BERT
distillation, with only about 28% parameters and about 31% inference time of
them. Moreover, TinyBERT with 6 layers performs on-par with its teacher
BERTBASE.

通过新的 Transformer 蒸馏方法和两阶段 TinyBERT 学习框架，可以有效地将大型 BERT 中的知识转移到小型 TinyBERT，从而在维持准确性的同时加速推理和减少模型大小，TinyBERT 在短语匹配任务的 GLUE 数据集上取得了 96.8% 以上的性能，模型大小约为 BERT 的 1/8，推理速度约为 BERT 的 1/10。