Transformer neural networks can exhibit a surprising capacity for in-context
learning (ICL) despite not being explicitly trained for it. Prior work has
provided a deeper understanding of how ICL emerges in transformers, e.g.
through the lens of mechanistic interpretability, Bayesian inference, or by
examining the distributional properties of training data. However, in each of
these cases, ICL is treated largely as a persistent phenomenon; namely, once
ICL emerges, it is assumed to persist asymptotically. Here, we show that the
emergence of ICL during transformer training is, in fact, often transient. We
train transformers on synthetic data designed so that both ICL and in-weights
learning (IWL) strategies can lead to correct predictions. We find that ICL
first emerges, then disappears and gives way to IWL, all while the training
loss decreases, indicating an asymptotic preference for IWL. The transient
nature of ICL is observed in transformers across a range of model sizes and
datasets, raising the question of how much to "overtrain" transformers when
seeking compact, cheaper-to-run models. We find that L2 regularization may
offer a path to more persistent ICL that removes the need for early stopping
based on ICL-style validation tasks. Finally, we present initial evidence that
ICL transience may be caused by competition between ICL and IWL circuits.

transformer 神经网络虽然没有明确训练用于上下文学习，但其仍具备令人惊讶的上下文学习能力。然而，研究发现 ICL 在训练过程中常常是短暂的，且 ICL 和 in-weights learning 之间存在竞争关系，使用 L2 正则化可能提供更持久的 ICL。