What computational structure are we building into large language models when we train them on next-token prediction? Here, we present evidence that this structure is given by the meta-dynamics of belief updating over hidden states of the data-generating process. Leveraging the theory of optimal prediction, we anticipate and then find that belief states are linearly represented in the residual stream of transformers, even in cases where the predicted belief state geometry has highly nontrivial fractal structure. We investigate cases where the belief state geometry is represented in the final residual stream or distributed across the residual streams of multiple layers, providing a framework to explain these observations. Furthermore we demonstrate that the inferred belief states contain information about the entire future, beyond the local next-token prediction that the transformers are explicitly trained on. Our work provides a framework connecting the structure of training data to the computational structure and representations that transformers use to carry out their behavior.

我们的研究显示，在训练大型语言模型进行下一个标记预测时，我们构建了一种计算结构，其表现为对数据生成过程隐藏状态的信念更新的元动力学。我们预测并发现，信念状态在变压器的残余流中线性表示，即使预测的信念状态具有高度复杂的分形结构。我们探讨了信念状态几何在最终残余流中或分布在多个层的残余流中的情况，提供了解释这些观察的框架。此外，我们证明了推断的信念状态包含关于整个未来的信息，超出变压器的显式训练目标——局部下一个标记预测。我们的工作提供了连接训练数据结构和变压器使用的计算结构和表示的框架。

在残差流中以Transformer表示信念状态几何