We propose pre-finetuning, an additional large-scale learning stage between
language model pre-training and fine-tuning. Pre-finetuning is massively
multi-task learning (around 50 datasets, over 4.8 million total labeled
examples), and is designed to encourage learning of representations that
generalize better to many different tasks. We show that pre-finetuning
consistently improves performance for pretrained discriminators (e.g.~RoBERTa)
and generation models (e.g.~BART) on a wide range of tasks (sentence
prediction, commonsense reasoning, MRC, etc.), while also significantly
improving sample efficiency during fine-tuning. We also show that large-scale
multi-tasking is crucial; pre-finetuning can hurt performance when few tasks
are used up until a critical point (usually above 15) after which performance
improves linearly in the number of tasks.

本文提出了预细调（pre-finetuning）的方法，是在语言模型预训练和微调之间增加一个大规模学习阶段，旨在促进学习出更好地适用于许多不同任务的表示。通过大规模的多任务学习，预细调在广泛的任务中显著提高了预训练鉴别器和生成模型的性能，并显著提高了微调的样本效率。