Dyna is a fundamental approach to model-based reinforcement learning (MBRL)
that interleaves planning, acting, and learning in an online setting. In the
most typical application of Dyna, the dynamics model is used to generate
one-step transitions from selected start states from the agent's history, which
are used to update the agent's value function or policy as if they were real
experiences. In this work, one-step Dyna was applied to several games from the
Arcade Learning Environment (ALE). We found that the model-based updates
offered surprisingly little benefit over simply performing more updates with
the agent's existing experience, even when using a perfect model. We
hypothesize that to get the most from planning, the model must be used to
generate unfamiliar experience. To test this, we experimented with the "shape"
of planning in multiple different concrete instantiations of Dyna, performing
fewer, longer rollouts, rather than many short rollouts. We found that planning
shape has a profound impact on the efficacy of Dyna for both perfect and
learned models. In addition to these findings regarding Dyna in general, our
results represent, to our knowledge, the first time that a learned dynamics
model has been successfully used for planning in the ALE, suggesting that Dyna
may be a viable approach to MBRL in the ALE and other high-dimensional
problems.

本论文研究了 Dyna（一种基于模型的强化学习）在多个游戏中的应用，发现规划的形式对 Dyna 的有效性有深刻影响，同时也是第一次成功地使用了手头有限的学习动态模型进行规划，该研究表明 Dyna 是解决高维度问题中的模型泛化能力问题的一种可行方法。