Goal-conditioned hierarchical reinforcement learning (GCHRL) provides a
promising approach to solving long-horizon tasks. Recently, its success has
been extended to more general settings by concurrently learning hierarchical
policies and subgoal representations. Although GCHRL possesses superior
exploration ability by decomposing tasks via subgoals, existing GCHRL methods
struggle in temporally extended tasks with sparse external rewards, since the
high-level policy learning relies on external rewards. As the high-level policy
selects subgoals in an online learned representation space, the dynamic change
of the subgoal space severely hinders effective high-level exploration. In this
paper, we propose a novel regularization that contributes to both stable and
efficient subgoal representation learning. Building upon the stable
representation, we design measures of novelty and potential for subgoals, and
develop an active hierarchical exploration strategy that seeks out new
promising subgoals and states without intrinsic rewards. Experimental results
show that our approach significantly outperforms state-of-the-art baselines in
continuous control tasks with sparse rewards.

本文提出了一种新的规范化方法来提高子目标表示的稳定性和效率，并设计了一种主动式分层探索策略来寻找没有内在奖励的新有前途的子目标和状态，实验结果表明，我们的方法在具有稀疏奖励的连续控制任务中显著优于最先进的基线算法。