Planning is a data efficient decision-making strategy where an agent selects
candidate actions by exploring possible future states. To simulate future
states when there is a high-dimensional action space, the knowledge of one's
decision making strategy must be used to limit the number of actions to be
explored. We refer to the model used to simulate one's decisions as the agent's
self-model. While self-models are implicitly used widely in conjunction with
world models to plan actions, it remains unclear how self-models should be
designed. Inspired by current reinforcement learning approaches and
neuroscience, we explore the benefits and limitations of using a distilled
policy network as the self-model. In such dual-policy agents, a model-free
policy and a distilled policy are used for model-free actions and planned
actions, respectively. Our results on a ecologically relevant, parametric
environment indicate that distilled policy network for self-model stabilizes
training, has faster inference than using model-free policy, promotes better
exploration, and could learn a comprehensive understanding of its own
behaviors, at the cost of distilling a new network apart from the model-free
policy.

本文研究了利用强化学习中的策略网络作为自模型来规划行动的决策方法，结果表明，相比于自由模型，这种方法训练更稳定，推理速度更快，探索性更好，能够得到一种全面的自我行为理解。