A promising way to improve the sample efficiency of reinforcement learning is model-based methods, in which many explorations and evaluations can happen in the learned models to save real-world samples. However, when the learned model has a non-negligible model error, sequential steps in the model are hard to be accurately evaluated, limiting the model's utilization. This paper proposes to alleviate this issue by introducing multi-step plans to replace multi-step actions for model-based RL. We employ the multi-step plan value estimation, which evaluates the expected discounted return after executing a sequence of action plans at a given state, and updates the policy by directly computing the multi-step policy gradient via plan value estimation. The new model-based reinforcement learning algorithm MPPVE (Model-based Planning Policy Learning with Multi-step Plan Value Estimation) shows a better utilization of the learned model and achieves a better sample efficiency than state-of-the-art model-based RL approaches.

本文提出了一种新的基于模型的强化学习算法 MPPVE（Model-based Planning Policy Learning with Multi-step Plan Value Estimation），通过引入多步计划来替换多步行动，采用多步计划价值估计来更新政策，从而更好地利用学习到的模型，实现比现有基于模型的强化学习方法更好的样本效率。

基于模型的强化学习模型——使用多步计划价值评估