Goal-Conditioned Reinforcement Learning (GCRL) can enable agents to
spontaneously set diverse goals to learn a set of skills. Despite the excellent
works proposed in various fields, reaching distant goals in temporally extended
tasks remains a challenge for GCRL. Current works tackled this problem by
leveraging planning algorithms to plan intermediate subgoals to augment GCRL.
Their methods need two crucial requirements: (i) a state representation space
to search valid subgoals, and (ii) a distance function to measure the
reachability of subgoals. However, they struggle to scale to high-dimensional
state space due to their non-compact representations. Moreover, they cannot
collect high-quality training data through standard GC policies, which results
in an inaccurate distance function. Both affect the efficiency and performance
of planning and policy learning. In the paper, we propose a goal-conditioned RL
algorithm combined with Disentanglement-based Reachability Planning (REPlan) to
solve temporally extended tasks. In REPlan, a Disentangled Representation
Module (DRM) is proposed to learn compact representations which disentangle
robot poses and object positions from high-dimensional observations in a
self-supervised manner. A simple REachability discrimination Module (REM) is
also designed to determine the temporal distance of subgoals. Moreover, REM
computes intrinsic bonuses to encourage the collection of novel states for
training. We evaluate our REPlan in three vision-based simulation tasks and one
real-world task. The experiments demonstrate that our REPlan significantly
outperforms the prior state-of-the-art methods in solving temporally extended
tasks.

我们提出了一种基于目标条件的强化学习算法，结合了解缠绕的可达性规划（REPlan），用于解决时间延展任务，在模拟和真实世界任务中，REPlan 显著优于之前最先进的方法。