Hierarchical reinforcement learning (HRL) has led to remarkable achievements
in diverse fields. However, existing HRL algorithms still cannot be applied to
real-world navigation tasks. These tasks require an agent to perform
safety-aware behaviors and interact with surrounding objects in dynamic
environments. In addition, an agent in these tasks should perform consistent
and structured exploration as they are long-horizon and have complex structures
with diverse objects and task-specific rules. Designing HRL agents that can
handle these challenges in real-world navigation tasks is an open problem. In
this paper, we propose imagination-augmented HRL (IAHRL), a new and general
navigation algorithm that allows an agent to learn safe and interactive
behaviors in real-world navigation tasks. Our key idea is to train a
hierarchical agent in which a high-level policy infers interactions by
interpreting behaviors imagined with low-level policies. Specifically, the
high-level policy is designed with a permutation-invariant attention mechanism
to determine which low-level policy generates the most interactive behavior,
and the low-level policies are implemented with an optimization-based behavior
planner to generate safe and structured behaviors following task-specific
rules. To evaluate our algorithm, we introduce five complex urban driving
tasks, which are among the most challenging real-world navigation tasks. The
experimental results indicate that our hierarchical agent performs safety-aware
behaviors and properly interacts with surrounding vehicles, achieving higher
success rates and lower average episode steps than baselines in urban driving
tasks.

通过引入想象增强的分层强化学习算法，我们设计了一种新的导航算法，使得智能体能够在真实世界的导航任务中学习安全和交互的行为，提高了成功率并减少了平均步骤的数量。