Overparameterized transformer networks have obtained state of the art results
in various natural language processing tasks, such as machine translation,
language modeling, and question answering. These models contain hundreds of
millions of parameters, necessitating a large amount of computation and making
them prone to overfitting. In this work, we explore LayerDrop, a form of
structured dropout, which has a regularization effect during training and
allows for efficient pruning at inference time. In particular, we show that it
is possible to select sub-networks of any depth from one large network without
having to finetune them and with limited impact on performance. We demonstrate
the effectiveness of our approach by improving the state of the art on machine
translation, language modeling, summarization, question answering, and language
understanding benchmarks. Moreover, we show that our approach leads to small
BERT-like models of higher quality compared to training from scratch or using
distillation.

本文介绍了一种新的结构化 dropout 形式 ——LayerDrop，该形式可在训练过程中实现正则化效果，在推理时允许有效的剪枝。作者通过实验证明了该方法对机器翻译、语言模型、文本摘要、问答和语言理解等基准测试的提升，并提出使用该方法可以得到更高质量的类 BERT 模型。