Hierarchical reinforcement learning has been a compelling approach for
achieving goal directed behavior over long sequences of actions. However, it
has been challenging to implement in realistic or open-ended environments. A
main challenge has been to find the right space of sub-goals over which to
instantiate a hierarchy. We present a novel approach where we use data from
humans solving these tasks to softly supervise the goal space for a set of long
range tasks in a 3D embodied environment. In particular, we use unconstrained
natural language to parameterize this space. This has two advantages: first, it
is easy to generate this data from naive human participants; second, it is
flexible enough to represent a vast range of sub-goals in human-relevant tasks.
Our approach outperforms agents that clone expert behavior on these tasks, as
well as HRL from scratch without this supervised sub-goal space. Our work
presents a novel approach to combining human expert supervision with the
benefits and flexibility of reinforcement learning.

我们提出了一种新颖的方法，利用人类在 3D 实体环境中解决任务时使用的无约束自然语言数据，通过软约束目标空间，对一组长期任务进行层次强化学习，从而实现在现实或开放环境中实现目标导向行为的挑战。