How do language models "think"? This paper formulates a probabilistic
cognitive model called bounded pragmatic speaker, which can characterize the
operation of different variants of language models. In particular, we show that
large language models fine-tuned with reinforcement learning from human
feedback (Ouyang et al., 2022) implements a model of thought that conceptually
resembles a fast-and-slow model (Kahneman, 2011). We discuss the limitations of
reinforcement learning from human feedback as a fast-and-slow model of thought
and propose directions for extending this framework. Overall, our work
demonstrates that viewing language models through the lens of cognitive
probabilistic modeling can offer valuable insights for understanding,
evaluating, and developing them.

本文通过概率认知模型解释了语言模型的运行方式，尤其是展示了通过强化学习从人类反馈中优化的大型语言模型实现了一种类似于 Kahneman 快慢思考模型的思维模型，并讨论了强化学习作为快慢思考模型的局限性，并提出了扩展此框架的方向。总的来说，我们的工作表明，通过认知概率建模的视角来理解、评估和发展语言模型可以提供有价值的见解。