Adapting large language models (LLMs) to unseen tasks with in-context
training samples without fine-tuning remains an important research problem. To
learn a robust LLM that adapts well to unseen tasks, multiple meta-training
approaches have been proposed such as MetaICL and MetaICT, which involve
meta-training pre-trained LLMs on a wide variety of diverse tasks. These
meta-training approaches essentially perform in-context multi-task fine-tuning
and evaluate on a disjointed test set of tasks. Even though they achieve
impressive performance, their goal is never to compute a truly general set of
parameters. In this paper, we propose MAML-en-LLM, a novel method for
meta-training LLMs, which can learn truly generalizable parameters that not
only perform well on disjointed tasks but also adapts to unseen tasks. We see
an average increase of 2% on unseen domains in the performance while a massive
4% improvement on adaptation performance. Furthermore, we demonstrate that
MAML-en-LLM outperforms baselines in settings with limited amount of training
data on both seen and unseen domains by an average of 2%. Finally, we discuss
the effects of type of tasks, optimizers and task complexity, an avenue barely
explored in meta-training literature. Exhaustive experiments across 7 task
settings along with two data settings demonstrate that models trained with
MAML-en-LLM outperform SOTA meta-training approaches.

我们提出了 MAML-en-LLM，一种新的元训练大型语言模型的方法，可以学习到真正可泛化的参数，不仅在不同任务上表现良好，还适应了未知任务，并且在性能和适应性能上得到了显著提升。