Recently, Language Models (LMs) instruction-tuned on multiple tasks, also known as multitask-prompted fine-tuning (MT), have shown the capability to generalize to unseen tasks. Previous work has shown that scaling the number of training tasks is the key component in making stronger MT LMs. In this work, we report an unexpected finding that an expert LM fine-tuned on just a single task can outperform an MT LM trained with 300+ different tasks on 11 different unseen datasets and on 13 datasets of the BIG-bench benchmark by a mean accuracy of 3.20% and 1.29%, respectively. This finding casts doubt on the previously held belief that simply scaling the number of tasks makes stronger MT LMs. Leveraging this finding, we further show that this distributed approach of training a separate expert LM per training task instead of a single MT LM for zero-shot inference possesses many benefits including (1) avoiding negative task transfer that often occurs during instruction tuning, (2) being able to continually learn new tasks without having to re-train on previous tasks to avoid catastrophic forgetting, and (3) showing compositional capabilities when merging individual experts together. The code is available at https://github.com/joeljang/ELM.

本文研究发现，相对于使用多个任务的multitask-prompted fine-tuning方法进行指导调整的语言模型，仅针对单个任务进行专家模型的fine-tuning能够使模型在11个不同的未知数据集上以及13个BIG-bench基准测试数据集上平均准确率分别提高3.20%和1.29%。此外，分别训练每个专家模型而不是单个MT LM进行zero-shot推断具有许多好处，包括避免负面任务转移，能够持续学习新任务而无需重新训练以避免灾难性遗忘以及显示在合并单个专家时具有组合能力。

训练专家语言模型带来的益处：比指令调整更胜一筹