Large-scale language model pretraining is a very successful form of self-supervised learning in natural language processing, but it is increasingly expensive to perform as the models and pretraining corpora have become larger over time. We propose NarrowBERT, a modified transformer encoder that increases the throughput for masked language model pretraining by more than $2\times$. NarrowBERT sparsifies the transformer model such that the self-attention queries and feedforward layers only operate on the masked tokens of each sentence during pretraining, rather than all of the tokens as with the usual transformer encoder. We also show that NarrowBERT increases the throughput at inference time by as much as $3.5\times$ with minimal (or no) performance degradation on sentence encoding tasks like MNLI. Finally, we examine the performance of NarrowBERT on the IMDB and Amazon reviews classification and CoNLL NER tasks and show that it is also comparable to standard BERT performance.

本文介绍了一种修改的transformer编码器--NarrowBERT，其通过使自注意力查询和前向层仅在预训练期间的屏蔽令牌上操作，从而增加了掩码语言模型预训练的吞吐量。 此外，本文还显示，与MNLI等句子编码任务相比，NarrowBERT在推理时间的吞吐量提高了多达3.5倍，性能降低最小（或没有），并且在IMDB，亚马逊评论分类和CoNLL NER任务上的NarrowBERT表现也与标准BERT相当。

NarrowBERT: 增强遮蔽语言模型的预训练和推理速度