Goal representation affects the performance of Hierarchical Reinforcement
Learning (HRL) algorithms by decomposing the complex learning problem into
easier subtasks. Recent studies show that representations that preserve
temporally abstract environment dynamics are successful in solving difficult
problems and provide theoretical guarantees for optimality. These methods
however cannot scale to tasks where environment dynamics increase in complexity
i.e. the temporally abstract transition relations depend on larger number of
variables. On the other hand, other efforts have tried to use spatial
abstraction to mitigate the previous issues. Their limitations include
scalability to high dimensional environments and dependency on prior knowledge.
In this paper, we propose a novel three-layer HRL algorithm that introduces,
at different levels of the hierarchy, both a spatial and a temporal goal
abstraction. We provide a theoretical study of the regret bounds of the learned
policies. We evaluate the approach on complex continuous control tasks,
demonstrating the effectiveness of spatial and temporal abstractions learned by
this approach.

通过引入空间和时间目标抽象的三层层次强化学习（HRL）算法提高目标表示性能，评估了该算法在复杂连续控制任务上学习到的空间和时间抽象的有效性以及遗憾边界的理论研究。

协调空间和时间抽象以实现目标表征

Reconciling Spatial and Temporal Abstractions for Goal Representation

Deep reinforcement learning (DRL) has shown incredible performance in
learning various tasks to the human level. However, unlike human perception,
current DRL models connect the entire low-level sensory input to the
state-action values rather than exploiting the relationship between and among
entities that constitute the sensory input. Because of this difference, DRL
needs vast amount of experience samples to learn. In this paper, we propose a
Multi-focus Attention Network (MANet) which mimics human ability to spatially
abstract the low-level sensory input into multiple entities and attend to them
simultaneously. The proposed method first divides the low-level input into
several segments which we refer to as partial states. After this segmentation,
parallel attention layers attend to the partial states relevant to solving the
task. Our model estimates state-action values using these attended partial
states. In our experiments, MANet attains highest scores with significantly
less experience samples. Additionally, the model shows higher performance
compared to the Deep Q-network and the single attention model as benchmarks.
Furthermore, we extend our model to attentive communication model for
performing multi-agent cooperative tasks. In multi-agent cooperative task
experiments, our model shows 20% faster learning than existing state-of-the-art
model.

本文提出了一种多焦点注意力网络 (Multi-focus Attention Network, MANet) 模型，通过模仿人类对低级感官输入进行空间抽象并同时关注它们的能力，将其分割成若干个部分状态后，利用并行的注意力层关注与任务有关的部分状态进行状态值估计；实验证明，MANet 在体验采样数显著减少的情况下取得了最高分，并相比 Deep Q-network 和单一注意力模型表现更好，同时在多智能体合作任务中，我们的模型比现有最先进模型的学习速度提高了 20%。