Recent advances in pre-training huge models on large amounts of text through self supervision have obtained state-of-the-art results in various natural language processing tasks. However, these huge and expensive models are difficult to use in practise for downstream tasks. Some recent efforts use knowledge distillation to compress these models. However, we see a gap between the performance of the smaller student models as compared to that of the large teacher. In this work, we leverage large amounts of in-domain unlabeled transfer data in addition to a limited amount of labeled training instances to bridge this gap. We show that simple RNN based student models even with hard distillation can perform at par with the huge teachers given the transfer set. The student performance can be further improved with soft distillation and leveraging teacher intermediate representations. We show that our student models can compress the huge teacher by up to 26x while still matching or even marginally exceeding the teacher performance in low-resource settings with small amount of labeled data.

该研究论文针对知识蒸馏在自然语言处理中预训练模型所遇到的问题，探讨利用领域内未标记的数据、有限的已标记数据和基于简单RNN模型的硬蒸馏方法解决师生模型间性能差异的问题。实验结果表明，使用软蒸馏和利用教师模型的中间表示，学生模型的性能可以进一步提高，而在低资源环境下，学生模型在保证与教师模型性能基本持平的情况下，可实现最多26倍的压缩比，针对多语言环境的扩展实验结果更是惊人。

用未标记的转移数据将BERT蒸馏为简单的神经网络