Operating in the real-world often requires agents to learn about a complex
environment and apply this understanding to achieve a breadth of goals. This
problem, known as goal-conditioned reinforcement learning (GCRL), becomes
especially challenging for long-horizon goals. Current methods have tackled
this problem by augmenting goal-conditioned policies with graph-based planning
algorithms. However, they struggle to scale to large, high-dimensional state
spaces and assume access to exploration mechanisms for efficiently collecting
training data. In this work, we introduce Successor Feature Landmarks (SFL), a
framework for exploring large, high-dimensional environments so as to obtain a
policy that is proficient for any goal. SFL leverages the ability of successor
features (SF) to capture transition dynamics, using it to drive exploration by
estimating state-novelty and to enable high-level planning by abstracting the
state-space as a non-parametric landmark-based graph. We further exploit SF to
directly compute a goal-conditioned policy for inter-landmark traversal, which
we use to execute plans to "frontier" landmarks at the edge of the explored
state space. We show in our experiments on MiniGrid and ViZDoom that SFL
enables efficient exploration of large, high-dimensional state spaces and
outperforms state-of-the-art baselines on long-horizon GCRL tasks.

本文介绍 Successor Feature Landmarks（SFL），它是用于大型、高维空间的探索的一个框架，该框架利用继承特征（SF）的能力来驱动探索，估计状态新颖性，并通过将状态空间抽象为基于非参数地标的图表，启用高级别规划，实现了 GCRL 任务上的超越表现。

基于后继特征标志的长视程目标导向强化学习

Successor Feature Landmarks for Long-Horizon Goal-Conditioned  Reinforcement Learning

We consider the problem of learning useful robotic skills from previously
collected offline data without access to manually specified rewards or
additional online exploration, a setting that is becoming increasingly
important for scaling robot learning by reusing past robotic data. In
particular, we propose the objective of learning a functional understanding of
the environment by learning to reach any goal state in a given dataset. We
employ goal-conditioned Q-learning with hindsight relabeling and develop
several techniques that enable training in a particularly challenging offline
setting. We find that our method can operate on high-dimensional camera images
and learn a variety of skills on real robots that generalize to previously
unseen scenes and objects. We also show that our method can learn to reach
long-horizon goals across multiple episodes through goal chaining, and learn
rich representations that can help with downstream tasks through pre-training
or auxiliary objectives. The videos of our experiments can be found at
this https URL

本文提出了使用自回归进行离线机器人技能学习，并通过先前的数据学习环境的形式理解，该方法可以处理高维相机图像，可以学习到许多技能，并通过一系列技术进行训练，此方法具有较强的泛化能力，可以在多个目标之间进行目标链接，通过预训练或辅助目标学习到丰富的表示。