To achieve autonomy in a priori unknown real-world scenarios, agents should
be able to: i) act from high-dimensional sensory observations (e.g., images),
ii) learn from past experience to adapt and improve, and iii) be capable of
long horizon planning. Classical planning algorithms (e.g. PRM, RRT) are
proficient at handling long-horizon planning. Deep learning based methods in
turn can provide the necessary representations to address the others, by
modeling statistical contingencies between observations. In this direction, we
introduce a general-purpose planning algorithm called PALMER that combines
classical sampling-based planning algorithms with learning-based perceptual
representations. For training these perceptual representations, we combine
Q-learning with contrastive representation learning to create a latent space
where the distance between the embeddings of two states captures how easily an
optimal policy can traverse between them. For planning with these perceptual
representations, we re-purpose classical sampling-based planning algorithms to
retrieve previously observed trajectory segments from a replay buffer and
restitch them into approximately optimal paths that connect any given pair of
start and goal states. This creates a tight feedback loop between
representation learning, memory, reinforcement learning, and sampling-based
planning. The end result is an experiential framework for long-horizon planning
that is significantly more robust and sample efficient compared to existing
methods.

为了实现先验未知的真实世界场景下的自治，我们介绍了一种称为 PALMER 的通用规划算法，将经典基于采样的规划算法与基于学习的感知表示相结合，从而获得更健壮、更高效的长视距规划。

PALMER：感知 - 动作循环，带有长视距规划记忆

PALMER: Perception-Action Loop with Memory for Long-Horizon Planning

We present PRM-RL, a hierarchical method for long-range navigation task
completion that combines sampling based path planning with reinforcement
learning (RL). The RL agents learn short-range, point-to-point navigation
policies that capture robot dynamics and task constraints without knowledge of
the large-scale topology. Next, the sampling-based planners provide roadmaps
which connect robot configurations that can be successfully navigated by the RL
agent. The same RL agents are used to control the robot under the direction of
the planning, enabling long-range navigation. We use the Probabilistic Roadmaps
(PRMs) for the sampling-based planner. The RL agents are constructed using
feature-based and deep neural net policies in continuous state and action
spaces. We evaluate PRM-RL, both in simulation and on-robot, on two navigation
tasks with non-trivial robot dynamics: end-to-end differential drive indoor
navigation in office environments, and aerial cargo delivery in urban
environments with load displacement constraints. Our results show improvement
in task completion over both RL agents on their own and traditional
sampling-based planners. In the indoor navigation task, PRM-RL successfully
completes up to 215 m long trajectories under noisy sensor conditions, and the
aerial cargo delivery completes flights over 1000 m without violating the task
constraints in an environment 63 million times larger than used in training.

PRM-RL 是一种利用采样路径规划结合强化学习实现远程导航的分层方法，其中 RL 代理通过学习短距离点到点导航策略，采用面向特征和深度神经网络及 PRMs 进行构建，并通过采样计划器提供的规划路径实现机器人的控制。PRM-RL 的应用结果表明，在室内和城市环境下，比起单独使用 RL 代理或传统的采样路径规划方法，PRM-RL 实现了任务完成度的显著提升，在噪声传感器条件下成功完成长达 215 米的轨迹，且实现了对 1,000 米长的空中货运的无违约任务达成。