We address the problem of learning hierarchical deep neural network policies
for reinforcement learning. In contrast to methods that explicitly restrict or
cripple lower layers of a hierarchy to force them to use higher-level
modulating signals, each layer in our framework is trained to directly solve
the task, but acquires a range of diverse strategies via a maximum entropy
reinforcement learning objective. Each layer is also augmented with latent
random variables, which are sampled from a prior distribution during the
training of that layer. The maximum entropy objective causes these latent
variables to be incorporated into the layer's policy, and the higher level
layer can directly control the behavior of the lower layer through this latent
space. Furthermore, by constraining the mapping from latent variables to
actions to be invertible, higher layers retain full expressivity: neither the
higher layers nor the lower layers are constrained in their behavior. Our
experimental evaluation demonstrates that we can improve on the performance of
single-layer policies on standard benchmark tasks simply by adding additional
layers, and that our method can solve more complex sparse-reward tasks by
learning higher-level policies on top of high-entropy skills optimized for
simple low-level objectives.

本文介绍了一种学习分层深度神经网络策略，通过最大熵强化学习目标训练各层解决任务，并通过潜在随机变量进行增广，从而实现多层次策略学习。对标准基准测试任务，通过增加多层，可以改善高层策略表现。对于简单低层次的目标，高熵技能的优化学习可以解决更复杂的稀疏奖励任务。