Large language models have the ability to generate text that mimics patterns
in their inputs. We introduce a simple Markov Chain sequence modeling task in
order to study how this in-context learning (ICL) capability emerges. In our
setting, each example is sampled from a Markov chain drawn from a prior
distribution over Markov chains. Transformers trained on this task form
\emph{statistical induction heads} which compute accurate next-token
probabilities given the bigram statistics of the context. During the course of
training, models pass through multiple phases: after an initial stage in which
predictions are uniform, they learn to sub-optimally predict using in-context
single-token statistics (unigrams); then, there is a rapid phase transition to
the correct in-context bigram solution. We conduct an empirical and theoretical
investigation of this multi-phase process, showing how successful learning
results from the interaction between the transformer's layers, and uncovering
evidence that the presence of the simpler unigram solution may delay formation
of the final bigram solution. We examine how learning is affected by varying
the prior distribution over Markov chains, and consider the generalization of
our in-context learning of Markov chains (ICL-MC) task to $n$-grams for $n >
2$.

大型语言模型具有模仿输入模式的能力。本文引入了一个简单的马尔可夫链序列建模任务，以研究这种上下文学习能力的形成。我们通过训练变压器模型来计算给定上下文的准确下一个标记的概率。通过实证和理论研究，本文揭示了成功学习是由变压器层之间的交互作用所造成的，并发现简单的单个标记统计解可能会延迟最终的双标记解的形成。我们还考虑了马尔可夫链的先验分布以及将上下文学习拓展到大于 2 的 n 元组。