We study the phenomenon of \textit{in-context learning} (ICL) exhibited by
large language models, where they can adapt to a new learning task, given a
handful of labeled examples, without any explicit parameter optimization. Our
goal is to explain how a pre-trained transformer model is able to perform ICL
under reasonable assumptions on the pre-training process and the downstream
tasks. We posit a mechanism whereby a transformer can achieve the following:
(a) receive an i.i.d. sequence of examples which have been converted into a
prompt using potentially-ambiguous delimiters, (b) correctly segment the prompt
into examples and labels, (c) infer from the data a \textit{sparse linear
regressor} hypothesis, and finally (d) apply this hypothesis on the given test
example and return a predicted label. We establish that this entire procedure
is implementable using the transformer mechanism, and we give sample complexity
guarantees for this learning framework. Our empirical findings validate the
challenge of segmentation, and we show a correspondence between our posited
mechanisms and observed attention maps for step (c).

本文研究了大型语言模型展示的 “上下文学习” 现象，并解释了预训练的 transformer 模型如何在合理的假设下执行上下文学习。我们推出了一种机制，使得 transformer 模型能够正确划分上下文，推断出稀疏线性回归假说，并应用此假说进行预测，在该学习框架中的样本复杂度保证。