Finetuning language models on a collection of datasets phrased as
instructions has been shown to improve model performance and generalization to
unseen tasks. In this paper we explore instruction finetuning with a particular
focus on (1) scaling the number of tasks, (2) scaling the model size, and (3)
finetuning on chain-of-thought data. We find that instruction finetuning with
the above aspects dramatically improves performance on a variety of model
classes (PaLM, T5, U-PaLM), prompting setups (zero-shot, few-shot, CoT), and
evaluation benchmarks (MMLU, BBH, TyDiQA, MGSM, open-ended generation). For
instance, Flan-PaLM 540B instruction-finetuned on 1.8K tasks outperforms PALM
540B by a large margin (+9.4% on average). Flan-PaLM 540B achieves
state-of-the-art performance on several benchmarks, such as 75.2% on five-shot
MMLU. We also publicly release Flan-T5 checkpoints, which achieve strong
few-shot performance even compared to much larger models, such as PaLM 62B.
Overall, instruction finetuning is a general method for improving the
performance and usability of pretrained language models.

本文研究了在指令集合中对语言模型进行微调以改善其性能及推广其应用的方法，重点探讨了任务规模的扩展、模型大小的扩展和链式推理数据的微调，发现通过上述方面的微调显著提高了包括 PaLM、T5、U-PaLM 在内的各种模型类别，在零样本、少样本和 CoT 环境中的表现，以及在多项基准测试中的表现。