Pretrained transformers exhibit the remarkable ability of in-context learning
(ICL): they can learn tasks from just a few examples provided in the prompt
without updating any weights. This raises a foundational question: can ICL
solve fundamentally $\textit{new}$ tasks that are very different from those
seen during pretraining? To probe this question, we examine ICL's performance
on linear regression while varying the diversity of tasks in the pretraining
dataset. We empirically demonstrate a $\textit{task diversity threshold}$ for
the emergence of ICL. Below this threshold, the pretrained transformer cannot
solve unseen regression tasks as it behaves like a Bayesian estimator with the
$\textit{non-diverse pretraining task distribution}$ as the prior. Beyond this
threshold, the transformer significantly outperforms this estimator; its
behavior aligns with that of ridge regression, corresponding to a Gaussian
prior over $\textit{all tasks}$, including those not seen during pretraining.
These results highlight that, when pretrained on data with task diversity
greater than the threshold, transformers $\textit{can}$ solve fundamentally new
tasks in-context. Importantly, this capability hinges on it deviating from the
Bayes optimal estimator with the pretraining distribution as the prior. This
study underscores, in a concrete example, the critical role of task diversity,
alongside data and model scale, in the emergence of ICL. Code is available at
this https URL

通过研究预训练数据集中任务多样性的不同，探究在不同任务多样性阈值下，在新的情境下可以有多大能力。结果表明，如果预训练的数据具有足够的多样性，那么可以在新任务中解决问题，但是这种能力依赖于违背 Bayes 最优估计并将 Gauss 先验分布中所有任务的能力。