In-context learning (ICL) has emerged as a particularly remarkable
characteristic of Large Language Models (LLM): given a pretrained LLM and an
observed dataset, LLMs can make predictions for new data points from the same
distribution without fine-tuning. Numerous works have postulated ICL as
approximately Bayesian inference, rendering this a natural hypothesis. In this
work, we analyse this hypothesis from a new angle through the martingale
property, a fundamental requirement of a Bayesian learning system for
exchangeable data. We show that the martingale property is a necessary
condition for unambiguous predictions in such scenarios, and enables a
principled, decomposed notion of uncertainty vital in trustworthy,
safety-critical systems. We derive actionable checks with corresponding theory
and test statistics which must hold if the martingale property is satisfied. We
also examine if uncertainty in LLMs decreases as expected in Bayesian learning
when more data is observed. In three experiments, we provide evidence for
violations of the martingale property, and deviations from a Bayesian scaling
behaviour of uncertainty, falsifying the hypothesis that ICL is Bayesian.

在这项工作中，我们通过鞅属性从一个新的角度分析了大语言模型（LLM）的背景学习（ICL）是否可以视为贝叶斯推断。我们提出了鞅属性作为满足交换数据的贝叶斯学习系统的基本要求，并且证明了它在可信的、安全关键系统中具有重要性，可以提供一个有原则的、分解的不确定性概念。我们推导了必须满足的可操作检查项，并提供了理论和测试统计学来验证鞅属性的满足。同时，我们还检验了当观测到更多数据时，LLM 中的不确定性是否按照贝叶斯学习的预期减少。通过三个实验，我们提供了违反鞅属性和不符合贝叶斯不确定性缩放行为的证据，从而证明了 ICL 不是贝叶斯的假设。