Prior work on language models (LMs) shows that training on a large number of
diverse tasks improves few-shot learning (FSL) performance on new tasks. We
take this to the extreme, automatically extracting 413,299 tasks from internet
tables - orders of magnitude more than the next-largest public datasets.
Finetuning on the resulting dataset leads to improved FSL performance on
Natural Language Processing (NLP) tasks, but not proportionally to dataset
scale. In fact, we find that narrow subsets of our dataset sometimes outperform
more diverse datasets. For example, finetuning on software documentation from
this http URL raises FSL performance by a mean of +7.5% on 52 downstream
tasks, which beats training on 40 human-curated NLP datasets (+6.7%).
Finetuning on various narrow datasets leads to similar broad improvements
across test tasks, suggesting that the gains are not from domain adaptation but
adapting to FSL in general. We do not observe clear patterns between the
datasets that lead to FSL gains, leaving open questions about why certain data
helps with FSL.

该研究发现将预训练语言模型 (LMs) 应用于许多具有多样性任务的训练可提高少量任务学习（FSL）模型的性能。作者从互联网表格中提取了 413,299 个任务并证明其中的一些数据集能更好地用于 NLP 任务的 FSL 模型中。