We present the first mechanistic evidence that model-free reinforcement learning agents can learn to plan. This is achieved by applying a methodology based on concept-based interpretability to a model-free agent in Sokoban -- a commonly used benchmark for studying planning. Specifically, we demonstrate that DRC, a generic model-free agent introduced by Guez et al. (2019), uses learned concept representations to internally formulate plans that both predict the long-term effects of actions on the environment and influence action selection. Our methodology involves: (1) probing for planning-relevant concepts, (2) investigating plan formation within the agent's representations, and (3) verifying that discovered plans (in the agent's representations) have a causal effect on the agent's behavior through interventions. We also show that the emergence of these plans coincides with the emergence of a planning-like property: the ability to benefit from additional test-time compute. Finally, we perform a qualitative analysis of the planning algorithm learned by the agent and discover a strong resemblance to parallelized bidirectional search. Our findings advance understanding of the internal mechanisms underlying planning behavior in agents, which is important given the recent trend of emergent planning and reasoning capabilities in LLMs through RL

本研究首次提供了无模型强化学习智能体能够进行规划的机制性证据，填补了相关领域的研究空白。通过对一个基于概念的可解释性方法的应用，我们展示了无模型智能体如何利用学习到的概念表示内部形成计划，从而预测行为对环境的长期影响并影响行动选择。我们的发现为理解智能体内部规划行为的机制提供了重要进展，具有推动新兴规划与推理能力研究的潜在影响。

无模型强化学习中的新兴规划解读