Model-based reinforcement learning (RL) has demonstrated remarkable successes
on a range of continuous control tasks due to its high sample efficiency. To
save the computation cost of conducting planning online, recent practices tend
to distill optimized action sequences into an RL policy during the training
phase. Although the distillation can incorporate both the foresight of planning
and the exploration ability of RL policies, the theoretical understanding of
these methods is yet unclear. In this paper, we extend the policy improvement
step of Soft Actor-Critic (SAC) by developing an approach to distill from
model-based planning to the policy. We then demonstrate that such an approach
of policy improvement has a theoretical guarantee of monotonic improvement and
convergence to the maximum value defined in SAC. We discuss effective design
choices and implement our theory as a practical algorithm -- Model-based
Planning Distilled to Policy (MPDP) -- that updates the policy jointly over
multiple future time steps. Extensive experiments show that MPDP achieves
better sample efficiency and asymptotic performance than both model-free and
model-based planning algorithms on six continuous control benchmark tasks in
MuJoCo.

通过开发一种从基于模型的规划到策略的蒸馏方法，我们扩展了软负责人 - 评论者算法（SAC）的策略改进步骤，并证明了这种方法在改进过程中具有单调性和收敛性的理论保证。