Large-scale language models have recently demonstrated impressive empirical
performance. Nevertheless, the improved results are attained at the price of
bigger models, more power consumption, and slower inference, which hinder their
applicability to low-resource (both memory and computation) platforms.
Knowledge distillation (KD) has been demonstrated as an effective framework for
compressing such big models. However, large-scale neural network systems are
prone to memorize training instances, and thus tend to make inconsistent
predictions when the data distribution is altered slightly. Moreover, the
student model has few opportunities to request useful information from the
teacher model when there is limited task-specific data available. To address
these issues, we propose MixKD, a data-agnostic distillation framework that
leverages mixup, a simple yet efficient data augmentation approach, to endow
the resulting model with stronger generalization ability. Concretely, in
addition to the original training examples, the student model is encouraged to
mimic the teacher's behavior on the linear interpolation of example pairs as
well. We prove from a theoretical perspective that under reasonable conditions
MixKD gives rise to a smaller gap between the generalization error and the
empirical error. To verify its effectiveness, we conduct experiments on the
GLUE benchmark, where MixKD consistently leads to significant gains over the
standard KD training, and outperforms several competitive baselines.
Experiments under a limited-data setting and ablation studies further
demonstrate the advantages of the proposed approach.

提出了一种利用 Mixup 数据扩充技术的数据不可知蒸馏框架 MixKD，大幅度降低了一些实践性问题的影响，提升了大规模语言模型的泛化能力，实现了比标准 KD 培训更好的性能表现，能用于低资源平台上的应用。