Recent curriculum Reinforcement Learning (RL) has shown notable progress in
solving complex tasks by proposing sequences of surrogate tasks. However, the
previous approaches often face challenges when they generate curriculum goals
in a high-dimensional space. Thus, they usually rely on manually specified goal
spaces. To alleviate this limitation and improve the scalability of the
curriculum, we propose a novel curriculum method that automatically defines the
semantic goal space which contains vital information for the curriculum
process, and suggests curriculum goals over it. To define the semantic goal
space, our method discretizes continuous observations via vector
quantized-variational autoencoders (VQ-VAE) and restores the temporal relations
between the discretized observations by a graph. Concurrently, ours suggests
uncertainty and temporal distance-aware curriculum goals that converges to the
final goals over the automatically composed goal space. We demonstrate that the
proposed method allows efficient explorations in an uninformed environment with
raw goal examples only. Also, ours outperforms the state-of-the-art curriculum
RL methods on data efficiency and performance, in various goal-reaching tasks
even with ego-centric visual inputs.

我们提出了一种新的课程方法，通过自动定义语义目标空间以及在其上提出课程目标来缓解以往方法在高维度空间中生成课程目标时遇到的挑战，并改善课程的可伸缩性。我们通过向量量化变分自动编码器 (VQ-VAE) 将连续观测离散化，并通过图形恢复离散观测之间的时序关系。同时，我们提出了考虑不确定性和时间距离的课程目标，这些目标能够收敛到自动组合的目标空间的最终目标。我们证明了该方法仅仅通过原始目标示例就能在未知环境中实现高效探索，且在各种目标达成任务中，甚至使用自我中心视觉输入时，该方法在数据效率和性能上都优于最先进的课程增强学习方法。