In recent years, pretrained models revolutionized the paradigm of natural
language understanding (NLU), where we append a randomly initialized
classification head after the pretrained backbone, e.g. BERT, and finetune the
whole model. As the pretrained backbone makes a major contribution to the
improvement, we naturally expect a good pretrained classification head can also
benefit the training. However, the final-layer output of the backbone, i.e. the
input of the classification head, will change greatly during finetuning, making
the usual head-only pretraining (LP-FT) ineffective. In this paper, we find
that parameter-efficient tuning makes a good classification head, with which we
can simply replace the randomly initialized heads for a stable performance
gain. Our experiments demonstrate that the classification head jointly
pretrained with parameter-efficient tuning consistently improves the
performance on 9 tasks in GLUE and SuperGLUE.

本文中，我们发现参数高效调整可以使良好的分类头，用它可以取代随机初始化的头以获得稳定的性能提高。我们的实验表明，与参数高效调整预训练的分类头共同作用，可以在 GLUE 和 SuperGLUE 的 9 个任务中持续改善性能。