Heavily overparameterized language models such as BERT, XLNet and T5 have
achieved impressive success in many NLP tasks. However, their high model
complexity requires enormous computation resources and extremely long training
time for both pre-training and fine-tuning. Many works have studied model
compression on large NLP models, but only focusing on reducing inference time
while still requiring an expensive training process. Other works use extremely
large batch sizes to shorten the pre-training time, at the expense of higher
computational resource demands. In this paper, inspired by the Early-Bird
Lottery Tickets recently studied for computer vision tasks, we propose
EarlyBERT, a general computationally-efficient training algorithm applicable to
both pre-training and fine-tuning of large-scale language models. By slimming
the self-attention and fully-connected sub-layers inside a transformer, we are
the first to identify structured winning tickets in the early stage of BERT
training. We apply those tickets towards efficient BERT training, and conduct
comprehensive pre-training and fine-tuning experiments on GLUE and SQuAD
downstream tasks. Our results show that EarlyBERT achieves comparable
performance to standard BERT, with 35~45% less training time. Code is available
at this https URL

本篇论文提出了 EarlyBERT，这是一种通用的计算高效的训练算法，可用于大规模语言模型的预训练和微调，通过对自注意和全连接子层进行精简，第一次在 BERT 训练的早期阶段中发现了结构优质的 winning tickets，实验结果表明，与标准 BERT 相比，EarlyBERT 能够在 35-45％的训练时间内实现相当的性能。