In-context learning is one of the surprising and useful features of large
language models. How it works is an active area of research. Recently, stylized
meta-learning-like setups have been devised that train these models on a
sequence of input-output pairs $(x, f(x))$ from a function class using the
language modeling loss and observe generalization to unseen functions from the
same class. One of the main discoveries in this line of research has been that
for several problems such as linear regression, trained transformers learn
algorithms for learning functions in context. However, the inductive biases of
these models resulting in this behavior are not clearly understood. A model
with unlimited training data and compute is a Bayesian predictor: it learns the
pretraining distribution. It has been shown that high-capacity transformers
mimic the Bayesian predictor for linear regression. In this paper, we show
empirical evidence of transformers exhibiting the behavior of this ideal
learner across different linear and non-linear function classes. We also extend
the previous setups to work in the multitask setting and verify that
transformers can do in-context learning in this setup as well and the Bayesian
perspective sheds light on this setting also. Finally, via the example of
learning Fourier series, we study the inductive bias for in-context learning.
We find that in-context learning may or may not have simplicity bias depending
on the pretraining data distribution.

本文通过多种线性和非线性函数类的实证观察，延伸了之前的研究，表明了 transformers 的理想学习者表现，并探究了其在 Bayesian 模型和多任务环境下的应用，还以傅里叶级数为例研究了其归纳偏差。