We introduce MetaICL (Meta-training for In-Context Learning), a new
meta-training framework for few-shot learning where a pretrained language model
is tuned to do in-context learning on a large set of training tasks. This
meta-training enables the model to more effectively learn a new task in context
at test time, by simply conditioning on a few training examples with no
parameter updates or task-specific templates. We experiment on a large, diverse
collection of tasks consisting of 142 NLP datasets including classification,
question answering, natural language inference, paraphrase detection and more,
across seven different meta-training/target splits. MetaICL outperforms a range
of baselines including in-context learning without meta-training and multi-task
learning followed by zero-shot transfer. We find that the gains are
particularly significant for target tasks that have domain shifts from the
meta-training tasks, and that using a diverse set of the meta-training tasks is
key to improvements. We also show that MetaICL approaches (and sometimes beats)
the performance of models fully finetuned on the target task, and outperforms
much bigger models with nearly 8x parameters. Finally, we show that MetaICL is
complementary to human-written instructions, and the best performance can be
achieved by combining both approaches.

MetaICL 是一种新的元训练框架，用于少样本学习，通过在大量训练任务上进行元训练，调整预训练模型进行上下文学习，从而在测试时仅通过少量训练例子进行条件化得到更好的学习效果，该框架在包括分类、问答、自然语言推理、释义检测等在内的 142 个 NLP 数据集上进行了实验，比没有元训练的上下文学习、多任务学习加零 - shot 转移等基线表现更好，在元训练任务与目标任务之间存在域变换时性能提升尤其显著。同时，MetaICL 方法和通常情况下全微调的模型表现相当甚至更优，甚至比参数近 8 倍的更大的模型表现更好，最后也证明了 MetaICL 可以和人工编写的指令相辅相成，两种方法相结合可以得到最佳的性能