Intelligent agents need to select long sequences of actions to solve complex
tasks. While humans easily break down tasks into subgoals and reach them
through millions of muscle commands, current artificial intelligence is limited
to tasks with horizons of a few hundred decisions, despite large compute
budgets. Research on hierarchical reinforcement learning aims to overcome this
limitation but has proven to be challenging, current methods rely on manually
specified goal spaces or subtasks, and no general solution exists. We introduce
Director, a practical method for learning hierarchical behaviors directly from
pixels by planning inside the latent space of a learned world model. The
high-level policy maximizes task and exploration rewards by selecting latent
goals and the low-level policy learns to achieve the goals. Despite operating
in latent space, the decisions are interpretable because the world model can
decode goals into images for visualization. Director outperforms exploration
methods on tasks with sparse rewards, including 3D maze traversal with a
quadruped robot from an egocentric camera and proprioception, without access to
the global position or top-down view that was used by prior work. Director also
learns successful behaviors across a wide range of environments, including
visual control, Atari games, and DMLab levels.

通过在一个学习好的世界模型的潜在空间 planning，我们引入了一种名叫 Director 的实用方法，直接从像素学习分层行为，无需手动指定目标空间或子任务，并在具有稀疏奖励的任务中表现超越了探索方法。