Large language models (LLMs) have shown the emergent capability of in-context
learning (ICL). One line of research has explained ICL as functionally
performing gradient descent. In this paper, we introduce a new way of
diagnosing whether ICL is functionally equivalent to gradient-based learning.
Our approach is based on the inverse frequency effect (IFE) -- a phenomenon in
which an error-driven learner is expected to show larger updates when trained
on infrequent examples than frequent ones. The IFE has previously been studied
in psycholinguistics because humans show this effect in the context of
structural priming (the tendency for people to produce sentence structures they
have encountered recently); the IFE has been used as evidence that human
structural priming must involve error-driven learning mechanisms. In our
experiments, we simulated structural priming within ICL and found that LLMs
display the IFE, with the effect being stronger in larger models. We conclude
that ICL is indeed a type of gradient-based learning, supporting the hypothesis
that a gradient component is implicitly computed in the forward pass during
ICL. Our results suggest that both humans and LLMs make use of gradient-based,
error-driven processing mechanisms.

大型语言模型（LLMs）已显示出上下文学习（ICL）的新兴能力。研究对 ICL 的解释之一是将其功能性视为梯度下降。本文介绍了一种诊断 ICL 是否与基于梯度的学习具有功能等价性的新方法。我们的方法基于反向频率效应（IFE）- 一种错误驱动的学习者在训练罕见示例而不是常见示例时预计会显示更大更新的现象。我们的实验证明，在 ICL 中模拟结构启动效应后，LLMs 显示 IFE，且在更大模型中效应更强。我们得出结论，ICL 确实是一种基于梯度的学习，支持正向传递过程中隐式计算梯度的假设。我们的结果表明，人类和 LLMs 都利用基于梯度和错误驱动的处理机制。