Hierarchical reinforcement learning is a promising approach that uses temporal abstraction to solve complex long horizon problems. However, simultaneously learning a hierarchy of policies is unstable as it is challenging to train higher-level policy when the lower-level primitive is non-stationary. In this paper, we propose a novel hierarchical algorithm by generating a curriculum of achievable subgoals for evolving lower-level primitives using reinforcement learning and imitation learning. The lower level primitive periodically performs data relabeling on a handful of expert demonstrations using our primitive informed parsing approach. We provide expressions to bound the sub-optimality of our method and develop a practical algorithm for hierarchical reinforcement learning. Since our approach uses a handful of expert demonstrations, it is suitable for most robotic control tasks. Experimental evaluation on complex maze navigation and robotic manipulation environments show that inducing hierarchical curriculum learning significantly improves sample efficiency, and results in efficient goal conditioned policies for solving temporally extended tasks.

本文提出了一种新的分层算法，通过生成可实现的子目标的课程来训练逐层递进的策略，通过在少量专家演示中使用基础解析方法进行数据重标注，得到了可以用于多数机器人控制任务的适用的逐层递进的强化学习算法，实验结果表明，课程学习可以显著提高分层强化学习的效率。

CRISP: 层次强化学习中课程引导的基本原语子目标预测