Humans can learn a new language task efficiently with only few examples, by
leveraging their knowledge obtained when learning prior tasks. In this paper,
we explore whether and how such cross-task generalization ability can be
acquired, and further applied to build better few-shot learners across diverse
NLP tasks. We introduce CrossFit, a problem setup for studying cross-task
generalization ability, which standardizes seen/unseen task partitions, data
access during different learning stages, and the evaluation protocols. To
instantiate different seen/unseen task partitions in CrossFit and facilitate
in-depth analysis, we present the NLP Few-shot Gym, a repository of 160 diverse
few-shot NLP tasks created from open-access NLP datasets and converted to a
unified text-to-text format. Our analysis reveals that the few-shot learning
ability on unseen tasks can be improved via an upstream learning stage using a
set of seen tasks. We also observe that the selection of upstream learning
tasks can significantly influence few-shot performance on unseen tasks, asking
further analysis on task similarity and transferability.

研究表明，通过利用学习先前任务时获取的知识，仅使用少量示例即可有效地学习新语言任务。本文探讨了跨任务泛化能力是否以及如何获得，并进一步应用于构建跨不同自然语言处理任务的更好的少样本学习器。我们引入了 CrossFit 问题设置来研究跨任务泛化能力，该设置标准化了已知 / 未知任务划分、不同学习阶段的数据访问和评估协议。通过我们的研究发现，使用一组已知任务的上游学习阶段可以提高对未知任务的少样本学习能力，同时选择好上游学习任务可以显著影响在未知任务上的少样本性能，并引发对任务相似性和可转移性的进一步研究。我们还提出了包括 160 个不同少样本自然语言处理任务的 NLP Few-shot Gym