This paper explores the elusive mechanism underpinning in-context learning in
Large Language Models (LLMs). Our work provides a novel perspective by
examining in-context learning via the lens of surface repetitions. We
quantitatively investigate the role of surface features in text generation, and
empirically establish the existence of \emph{token co-occurrence
reinforcement}, a principle that strengthens the relationship between two
tokens based on their contextual co-occurrences. By investigating the dual
impacts of these features, our research illuminates the internal workings of
in-context learning and expounds on the reasons for its failures. This paper
provides an essential contribution to the understanding of in-context learning
and its potential limitations, providing a fresh perspective on this exciting
capability.

这篇论文通过检验表面重复现象的视角，定量地研究了文本生成中的表面特征对上下文学习的作用，并实证了一种被称为 “令牌共现强化” 的原则，强调了两个令牌在上下文共现下的关系。通过研究这些特征的双重影响，我们的研究揭示了上下文学习的内部机制和其失败的原因。这篇论文对于理解上下文学习及其潜在限制作出了重要贡献，并为这一令人兴奋的能力提供了新的视角。