The use of large transformer-based models such as BERT, GPT, and T5 has led
to significant advancements in natural language processing. However, these
models are computationally expensive, necessitating model compression
techniques that reduce their size and complexity while maintaining accuracy.
This project investigates and applies knowledge distillation for BERT model
compression, specifically focusing on the TinyBERT student model. We explore
various techniques to improve knowledge distillation, including experimentation
with loss functions, transformer layer mapping methods, and tuning the weights
of attention and representation loss and evaluate our proposed techniques on a
selection of downstream tasks from the GLUE benchmark. The goal of this work is
to improve the efficiency and effectiveness of knowledge distillation, enabling
the development of more efficient and accurate models for a range of natural
language processing tasks.

本研究使用了 Transformer-based 模型（如 BERT、GPT 和 T5），并进行了知识蒸馏来进行模型压缩，特别关注 TinyBERT 学生模型。通过实验不同的损失函数、Transformer 层映射方法和注意力和表示损失的权重调整，评估了提出的方法在 GLUE 基准测试的若干下游任务上的效果，旨在提高知识蒸馏技术的效率和准确性，为各种自然语言处理任务的开发提供更高效和准确的模型。

BERT 模型的知识蒸馏改进：损失函数、映射方法和权重调整

Improving Knowledge Distillation for BERT Models: Loss Functions,  Mapping Methods, and Weight Tuning

Despite the effectiveness of utilizing the BERT model for document ranking,
the high computational cost of such approaches limits their uses. To this end,
this paper first empirically investigates the effectiveness of two knowledge
distillation models on the document ranking task. In addition, on top of the
recently proposed TinyBERT model, two simplifications are proposed. Evaluations
on two different and widely-used benchmarks demonstrate that Simplified
TinyBERT with the proposed simplifications not only boosts TinyBERT, but also
significantly outperforms BERT-Base when providing 15$\times$ speedup.

该研究比较了两种知识蒸馏模型在文档排序任务中的有效性并对 TinyBERT 模型进行了两种简化，结果显示出这些简化不仅能够提升 TinyBERT 的性能，而且还可以在提供 15 倍速度提升的同时显著优于 BERT-Base。

简洁 TinyBERT：文档检索的知识蒸馏

Simplified TinyBERT: Knowledge Distillation for Document Retrieval

Language model pre-training, such as BERT, has significantly improved the
performances of many natural language processing tasks. However, pre-trained
language models are usually computationally expensive, so it is difficult to
efficiently execute them on resource-restricted devices. To accelerate
inference and reduce model size while maintaining accuracy, we first propose a
novel Transformer distillation method that is specially designed for knowledge
distillation (KD) of the Transformer-based models. By leveraging this new KD
method, the plenty of knowledge encoded in a large teacher BERT can be
effectively transferred to a small student Tiny-BERT. Then, we introduce a new
two-stage learning framework for TinyBERT, which performs Transformer
distillation at both the pretraining and task-specific learning stages. This
framework ensures that TinyBERT can capture he general-domain as well as the
task-specific knowledge in BERT.
TinyBERT with 4 layers is empirically effective and achieves more than 96.8%
the performance of its teacher BERTBASE on GLUE benchmark, while being 7.5x
smaller and 9.4x faster on inference. TinyBERT with 4 layers is also
significantly better than 4-layer state-of-the-art baselines on BERT
distillation, with only about 28% parameters and about 31% inference time of
them. Moreover, TinyBERT with 6 layers performs on-par with its teacher
BERTBASE.

通过新的 Transformer 蒸馏方法和两阶段 TinyBERT 学习框架，可以有效地将大型 BERT 中的知识转移到小型 TinyBERT，从而在维持准确性的同时加速推理和减少模型大小，TinyBERT 在短语匹配任务的 GLUE 数据集上取得了 96.8% 以上的性能，模型大小约为 BERT 的 1/8，推理速度约为 BERT 的 1/10。