We propose the Thinker algorithm, a novel approach that enables reinforcement
learning agents to autonomously interact with and utilize a learned world
model. The Thinker algorithm wraps the environment with a world model and
introduces new actions designed for interacting with the world model. These
model-interaction actions enable agents to perform planning by proposing
alternative plans to the world model before selecting a final action to execute
in the environment. This approach eliminates the need for hand-crafted planning
algorithms by enabling the agent to learn how to plan autonomously and allows
for easy interpretation of the agent's plan with visualization. We demonstrate
the algorithm's effectiveness through experimental results in the game of
Sokoban and the Atari 2600 benchmark, where the Thinker algorithm achieves
state-of-the-art performance and competitive results, respectively.
Visualizations of agents trained with the Thinker algorithm demonstrate that
they have learned to plan effectively with the world model to select better
actions. The algorithm's generality opens a new research direction on how a
world model can be used in reinforcement learning and how planning can be
seamlessly integrated into an agent's decision-making process.

我们提出了 Thinker 算法，这是一种新颖的方法，使强化学习代理能够与学习的世界模型自主交互并利用它。Thinker 算法通过在环境周围包装一个世界模型，并引入专门用于与世界模型交互的新动作，使代理能够通过向世界模型提议替代方案来进行规划，然后选择一个最终动作在环境中执行。该算法的有效性通过在 Sokoban 游戏和 Atari 2600 基准测试中的实验结果得到了证明，其中 Thinker 算法分别实现了最先进的性能和有竞争力的结果。使用 Thinker 算法训练的代理的可视化结果表明，它们已经学会了有效地利用世界模型进行规划，以选择更好的动作。该算法的普遍性为如何将世界模型用于强化学习以及如何将规划无缝集成到代理的决策过程中打开了一个新的研究方向。