In recent times, BERT-based models have been extremely successful in solving
a variety of natural language processing (NLP) tasks such as reading
comprehension, natural language inference, sentiment analysis, etc. All
BERT-based architectures have a self-attention block followed by a block of
intermediate layers as the basic building component. However, a strong
justification for the inclusion of these intermediate layers remains missing in
the literature. In this work we investigate the importance of intermediate
layers on the overall network performance of downstream tasks. We show that
reducing the number of intermediate layers and modifying the architecture for
BERT-BASE results in minimal loss in fine-tuning accuracy for downstream tasks
while decreasing the number of parameters and training time of the model.
Additionally, we use centered kernel alignment and probing linear classifiers
to gain insight into our architectural modifications and justify that removal
of intermediate layers has little impact on the fine-tuned accuracy.

本文探讨了 BERT 模型中中间层对于下游任务的影响，证明减少中间层数量和修改模型结构对下游任务的微调准确性影响很小，同时可以减少模型参数数量和训练时间。通过核心对齐和探究性线性分类器验证了去除中间层对于微调精度的影响微小。