Despite the recent success of multi-task learning and transfer learning for
natural language processing (NLP), few works have systematically studied the
effect of scaling up the number of tasks during pre-training. Towards this
goal, this paper introduces ExMix (Extreme Mixture): a massive collection of
107 supervised NLP tasks across diverse domains and task-families. Using ExMix,
we study the effect of multi-task pre-training at the largest scale to date,
and analyze co-training transfer amongst common families of tasks. Through this
analysis, we show that manually curating an ideal set of tasks for multi-task
pre-training is not straightforward, and that multi-task scaling can vastly
improve models on its own. Finally, we propose ExT5: a model pre-trained using
a multi-task objective of self-supervised span denoising and supervised ExMix.
Via extensive experiments, we show that ExT5 outperforms strong T5 baselines on
SuperGLUE, GEM, Rainbow, Closed-Book QA tasks, and several tasks outside of
ExMix. ExT5 also significantly improves sample efficiency while pre-training.

本文通过 ExMix 集合超过 107 个具有不同领域和任务族的监督自然语言处理任务，进行了 NLP 任务数量的缩放对模型多任务预训练效果的系统研究，并分析了任务家族间的共同训练迁移。其中，本文证明了手动筛选多任务预训练理想任务集并不容易，而多任务缩放本身可以大幅改进模型性能；同时，本文还提出 ExT5，通过超过 107 个 NLP 任务的多任务自监督 span 去噪和 ExMix 的监督多任务目标进行预训练，证明了 ExT5 在 SuperGLUE、GEM、Rainbow、闭卷 QA 任务和一些 ExMix 任务之外的任务上均优于强基线模型 T5，并显著提高了预训练的样本效率。