Large language models have become a vital component in modern NLP, achieving
state of the art performance in a variety of tasks. However, they are often
inefficient for real-world deployment due to their expensive inference costs.
Knowledge distillation is a promising technique to improve their efficiency
while retaining most of their effectiveness. In this paper, we reproduce,
compare and analyze several representative methods for task-agnostic
(general-purpose) distillation of Transformer language models. Our target of
study includes Output Distribution (OD) transfer, Hidden State (HS) transfer
with various layer mapping strategies, and Multi-Head Attention (MHA) transfer
based on MiniLMv2. Through our extensive experiments, we study the
effectiveness of each method for various student architectures in both
monolingual (English) and multilingual settings. Overall, we show that MHA
transfer based on MiniLMv2 is generally the best option for distillation and
explain the potential reasons behind its success. Moreover, we show that HS
transfer remains as a competitive baseline, especially under a sophisticated
layer mapping strategy, while OD transfer consistently lags behind other
approaches. Findings from this study helped us deploy efficient yet effective
student models for latency-critical applications.

通过多项实验研究，我们发现基于 MiniLMv2 的 Multi-Head Attention 转移是蒸馏中更优选的方法，并解释了其成功的可能原因。此外，我们还发现基于 Hidden State 转移的方法在精妙的层映射策略下依然是竞争性的基准，而 Output Distribution 转移一直落后于其他方法。研究结果帮助我们为延迟关键应用部署高效且有效的学生模型。