Foundation models exhibit significant capabilities in decision-making and
logical deductions. Nonetheless, a continuing discourse persists regarding
their genuine understanding of the world as opposed to mere stochastic mimicry.
This paper meticulously examines a simple transformer trained for Othello,
extending prior research to enhance comprehension of the emergent world model
of Othello-GPT. The investigation reveals that Othello-GPT encapsulates a
linear representation of opposing pieces, a factor that causally steers its
decision-making process. This paper further elucidates the interplay between
the linear world representation and causal decision-making, and their
dependence on layer depth and model complexity. We have made the code public.

这篇论文通过细致的研究一个经过训练的简单变压器模型在奥赛罗游戏中的应用，扩展先前的研究以提升对奥赛罗 - GPT 应用中新兴世界模型的理解，揭示了奥赛罗 - GPT 所包含的对立棋子的线性表达，在决策过程中起着因果驱动作用，并进一步阐述了线性世界表达和因果决策之间的相互作用，以及它们对层深和模型复杂度的依赖关系。我们已将代码公开。