Solving complex, temporally-extended tasks is a long-standing problem in
reinforcement learning (RL). We hypothesize that one critical element of
solving such problems is the notion of compositionality. With the ability to
learn concepts and sub-skills that can be composed to solve longer tasks, i.e.
hierarchical RL, we can acquire temporally-extended behaviors. However,
acquiring effective yet general abstractions for hierarchical RL is remarkably
challenging. In this paper, we propose to use language as the abstraction, as
it provides unique compositional structure, enabling fast learning and
combinatorial generalization, while retaining tremendous flexibility, making it
suitable for a variety of problems. Our approach learns an
instruction-following low-level policy and a high-level policy that can reuse
abstractions across tasks, in essence, permitting agents to reason using
structured language. To study compositional task learning, we introduce an
open-source object interaction environment built using the MuJoCo physics
engine and the CLEVR engine. We find that, using our approach, agents can learn
to solve to diverse, temporally-extended tasks such as object sorting and
multi-object rearrangement, including from raw pixel observations. Our analysis
reveals that the compositional nature of language is critical for learning
diverse sub-skills and systematically generalizing to new sub-skills in
comparison to non-compositional abstractions that use the same supervision.

本文提出了使用语言作为抽象概念的层次强化学习方法，通过学习基本概念和子技能来解决时间延长的任务，构建了开源对象交互环境，证明了采用这种方法，代理可以从原始像素观察中学习解决多样化的任务，语言的组合性质使得学习多种子技能变得关键。