Brown et al. (2020) famously introduced the phenomenon of in-context learning
in large language models (LLMs). We establish the existence of a phenomenon we
call $\textbf{meta-out-of-context learning (meta-OCL)}$ via carefully designed
synthetic experiments with LLMs. Our results suggest that meta-OCL leads LLMs
to more readily "internalize" the semantic content of text that is, or appears
to be, broadly useful (such as true statements, or text from authoritative
sources) and use it in appropriate circumstances. We further demonstrate
meta-OCL in a synthetic computer vision setting, and propose two hypotheses for
the emergence of meta-OCL: one relying on the way models store knowledge in
their parameters, and another suggesting that the implicit gradient alignment
bias of gradient-descent-based optimizers may be responsible. Finally, we
reflect on what our results might imply about capabilities of future AI
systems, and discuss potential risks. Our code can be found at
this https URL .

通过合成实验，我们发现了被称为 meta-OCL 的现象，该现象使 LLMs 更容易 “内部化” 广泛有用的文本的语义内容，并在适当的情况下使用它。我们在合成计算机视觉环境中进一步证明了 meta-OCL，并提出了两个假设：一个依赖模型在其参数中存储知识的方式，另一个暗示基于梯度下降优化器的隐式梯度对齐偏差可能是其原因。最后，我们思考了我们的研究结果对未来人工智能系统能力的暗示，并讨论了潜在的风险。