This paper investigates a new approach to model-based reinforcement learning using background planning: mixing (approximate) dynamic programming updates and model-free updates, similar to the Dyna architecture. Background planning with learned models is often worse than model-free alternatives, such as Double DQN, even though the former uses significantly more memory and computation. The fundamental problem is that learned models can be inaccurate and often generate invalid states, especially when iterated many steps. In this paper, we avoid this limitation by constraining background planning to a set of (abstract) subgoals and learning only local, subgoal-conditioned models. This goal-space planning (GSP) approach is more computationally efficient, naturally incorporates temporal abstraction for faster long-horizon planning and avoids learning the transition dynamics entirely. We show that our GSP algorithm can propagate value from an abstract space in a manner that helps a variety of base learners learn significantly faster in different domains.

本文提出了一种新的基于模型的强化学习方法，使用背景规划来混合（近似）动态规划和无模型更新，以解决学习模型的不准确性和生成无效状态的问题，并通过约束背景规划到一组（抽象的）子目标和只学习局部的、以子目标为条件的模型来提高计算效率，自然地包含时间抽象以实现更快的长期规划，并完全避免学习转换动力学，展示了该方法在不同领域中可以帮助各种基础学习者更快地学习。

在线强化学习中的规划的新视角