One of the gnarliest challenges in reinforcement learning (RL) is exploration
that scales to vast domains, where novelty-, or coverage-seeking behaviour
falls short. Goal-directed, purposeful behaviours are able to overcome this,
but rely on a good goal space. The core challenge in goal discovery is finding
the right balance between generality (not hand-crafted) and tractability
(useful, not too many). Our approach explicitly seeks the middle ground,
enabling the human designer to specify a vast but meaningful proto-goal space,
and an autonomous discovery process to refine this to a narrower space of
controllable, reachable, novel, and relevant goals. The effectiveness of
goal-conditioned exploration with the latter is then demonstrated in three
challenging environments.

本研究基于强化学习，通过在人工设计产生的广泛目标空间中寻找可控、可达、新颖和相关目标的自主发现过程，弥补探索广域领域中新奇性和涵盖性行为不足的问题，并在三种具有挑战性的环境中证明了目标导向的探索的有效性。

通过修剪原型目标扩展目标导向探索的规模

Scaling Goal-based Exploration via Pruning Proto-goals

We train a single, goal-conditioned policy that can solve many robotic
manipulation tasks, including tasks with previously unseen goals and objects.
We rely on asymmetric self-play for goal discovery, where two agents, Alice and
Bob, play a game. Alice is asked to propose challenging goals and Bob aims to
solve them. We show that this method can discover highly diverse and complex
goals without any human priors. Bob can be trained with only sparse rewards,
because the interaction between Alice and Bob results in a natural curriculum
and Bob can learn from Alice's trajectory when relabeled as a goal-conditioned
demonstration. Finally, our method scales, resulting in a single policy that
can generalize to many unseen tasks such as setting a table, stacking blocks,
and solving simple puzzles. Videos of a learned policy is available at
this https URL

通过不对人类先验假设的依赖，我们可以通过非对称自我博弈的方法训练出只需稀疏奖励的 Bob，并且他可以从 Alice 的轨迹中学习，以实现目标发现和机器人操作的一体化控制。