We develop a metalearning approach for learning hierarchically structured
policies, improving sample efficiency on unseen tasks through the use of shared
primitives---policies that are executed for large numbers of timesteps.
Specifically, a set of primitives are shared within a distribution of tasks,
and are switched between by task-specific policies. We provide a concrete
metric for measuring the strength of such hierarchies, leading to an
optimization problem for quickly reaching high reward on unseen tasks. We then
present an algorithm to solve this problem end-to-end through the use of any
off-the-shelf reinforcement learning method, by repeatedly sampling new tasks
and resetting task-specific policies. We successfully discover meaningful motor
primitives for the directional movement of four-legged robots, solely by
interacting with distributions of mazes. We also demonstrate the
transferability of primitives to solve long-timescale sparse-reward obstacle
courses, and we enable 3D humanoid robots to robustly walk and crawl with the
same policy.

本文提出了一种元学习方法，通过使用共享的基元，改进对未见过任务样本的样本效率，并提供了衡量其效果的具体指标；同时，通过重置任务特定策略并使用任何现成的强化学习方法，我们实现了这一方法的端到端解决方案，成功地在用四条腿的机器人互动的迷宫分布中发现了有意义的运动原语，并且将原语应用于解决长时间尺度的稀疏奖励障碍航线问题，从而使三维人形机器人能够稳健地行走和爬行。