In order to understand the in-context learning phenomenon, recent works have
adopted a stylized experimental framework and demonstrated that Transformers
can learn gradient-based learning algorithms for various classes of real-valued
functions. However, the limitations of Transformers in implementing learning
algorithms, and their ability to learn other forms of algorithms are not well
understood. Additionally, the degree to which these capabilities are confined
to attention-based models is unclear. Furthermore, it remains to be seen
whether the insights derived from these stylized settings can be extrapolated
to pretrained Large Language Models (LLMs). In this work, we take a step
towards answering these questions by demonstrating the following: (a) On a
test-bed with a variety of Boolean function classes, we find that Transformers
can nearly match the optimal learning algorithm for 'simpler' tasks, while
their performance deteriorates on more 'complex' tasks. Additionally, we find
that certain attention-free models perform (almost) identically to Transformers
on a range of tasks. (b) When provided a teaching sequence, i.e. a set of
examples that uniquely identifies a function in a class, we show that
Transformers learn more sample-efficiently. Interestingly, our results show
that Transformers can learn to implement two distinct algorithms to solve a
single task, and can adaptively select the more sample-efficient algorithm
depending on the sequence of in-context examples. (c) Lastly, we show that
extant LLMs, e.g. LLaMA-2, GPT-4, can compete with nearest-neighbor baselines
on prediction tasks that are guaranteed to not be in their training set.

为了理解上下文学习现象，最近的研究采用了一个简化的实验框架，并证明了 Transformer 可以学习各种实值函数的基于梯度的学习算法。然而，Transformer 在实现学习算法方面的局限性以及其学习其他形式算法的能力尚不明确，而这些能力在基于注意力模型中是否有限制也不清楚。此外，还需要进一步研究这些简化设置所得出的见解是否可以推广到预训练的大语言模型（LLMs）。在本研究中，我们通过以下方式来一步步回答这些问题：（a）在一个包含各种布尔函数类的测试集上，我们发现 Transformer 在更简单的任务上几乎可以与最佳学习算法相匹配，但在更复杂的任务上性能下降。此外，我们发现某些无注意力模型在一系列任务上与 Transformer 表现（几乎）一致；（b）当提供一个教学序列，即一组通过示例唯一标识一个函数类的示例时，我们发现 Transformer 学习起来更加高效。有趣的是，我们的结果表明，Transformer 可以学习实现两个不同的算法来解决一个任务，并且可以根据上下文示例的顺序自适应地选择更加高效的算法；（c）最后，我们展示了 LLMs（如 LLaMA-2、GPT-4）可以与最近邻基线在保证不在其训练集中的预测任务上竞争。