This paper investigates a new approach to model-based reinforcement learning using background planning: mixing (approximate) dynamic programming updates and model-free updates, similar to the Dyna architecture. Background planning with learned models is often worse than model-free alternatives, such as Double DQN, even though the former uses significantly more memory and computation. The fundamental problem is that learned models can be inaccurate and often generate invalid states, especially when iterated many steps. In this paper, we avoid this limitation by constraining background planning to a set of (abstract) subgoals and learning only local, subgoal-conditioned models. This goal-space planning (GSP) approach is more computationally efficient, naturally incorporates temporal abstraction for faster long-horizon planning and avoids learning the transition dynamics entirely. We show that our GSP algorithm can learn significantly faster than a Double DQN baseline in a variety of situations.

本文介绍了一种新的基于模型的强化学习方法来使用背景计划：将（近似）动态规划更新和无模型更新混合，类似于Dyna架构。我们通过将背景规划限制在一组（抽象）子目标上，并仅学习本地的、子目标条件模型的方法来避免高内存和计算使用率的不足，并证明了我们的GSP算法在各种情况下可以比Double DQN基线学习得更快。

子目标模型的目标空间规划