Pre-trained Language Models (LMs) have become an integral part of Natural
Language Processing (NLP) in recent years, due to their superior performance in
downstream applications. In spite of this resounding success, the usability of
LMs is constrained by computational and time complexity, along with their
increasing size; an issue that has been referred to as `overparameterisation'.
Different strategies have been proposed in the literature to alleviate these
problems, with the aim to create effective compact models that nearly match the
performance of their bloated counterparts with negligible performance losses.
One of the most popular techniques in this area of research is model
distillation. Another potent but underutilised technique is cross-layer
parameter sharing. In this work, we combine these two strategies and present
MiniALBERT, a technique for converting the knowledge of fully parameterised LMs
(such as BERT) into a compact recursive student. In addition, we investigate
the application of bottleneck adapters for layer-wise adaptation of our
recursive student, and also explore the efficacy of adapter tuning for
fine-tuning of compact models. We test our proposed models on a number of
general and biomedical NLP tasks to demonstrate their viability and compare
them with the state-of-the-art and other existing compact models. All the codes
used in the experiments are available at
this https URL. Our pre-trained compact models
can be accessed from this https URL.

本文介绍 MiniALBERT，一种将已有的完全参数化的语言模型转换为压缩递归模型的技术，探讨了论文中提出的模型蒸馏、跨层参数共享、瓶颈适配器等技术对压缩模型的 fine-tuning 效果以及在多种领域的实验结果。