Go-Explore achieved breakthrough performance on challenging reinforcement
learning (RL) tasks with sparse rewards. The key insight of Go-Explore was that
successful exploration requires an agent to first return to an interesting
state ('Go'), and only then explore into unknown terrain ('Explore'). We refer
to such exploration after a goal is reached as 'post-exploration'. In this
paper, we present a clear ablation study of post-exploration in a general
intrinsically motivated goal exploration process (IMGEP) framework, that the
Go-Explore paper did not show. We study the isolated potential of
post-exploration, by turning it on and off within the same algorithm under both
tabular and deep RL settings on both discrete navigation and continuous control
tasks. Experiments on a range of MiniGrid and Mujoco environments show that
post-exploration indeed helps IMGEP agents reach more diverse states and boosts
their performance. In short, our work suggests that RL researchers should
consider to use post-exploration in IMGEP when possible since it is effective,
method-agnostic and easy to implement.

该论文研究了强化学习领域中的一种新方法 “Go-Explore”，通过后探索 (post-exploration) 的方式探明未知状态，通过试验在 MiniGrid 和 Mujoco 环境下得出结论，该方法具有通用性，易于实现，RL 研究人员可考虑在 intrinsically motivated goal exploration process 中采用 post-exploration。

先 Go，后 Post-Exploration：后探索在内在动机中的益处

First Go, then Post-Explore: the Benefits of Post-Exploration in Intrinsic Motivation

Go-Explore achieved breakthrough performance on challenging reinforcement
learning (RL) tasks with sparse rewards. The key insight of Go-Explore was that
successful exploration requires an agent to first return to an interesting
state ('Go'), and only then explore into unknown terrain ('Explore'). We refer
to such exploration after a goal is reached as 'post-exploration'. In this
paper we present a systematic study of post-exploration, answering open
questions that the Go-Explore paper did not answer yet. First, we study the
isolated potential of post-exploration, by turning it on and off within the
same algorithm. Subsequently, we introduce new methodology to adaptively decide
when to post-explore and for how long to post-explore. Experiments on a range
of MiniGrid environments show that post-exploration indeed boosts performance
(with a bigger impact than tuning regular exploration parameters), and this
effect is further enhanced by adaptively deciding when and for how long to
post-explore. In short, our work identifies adaptive post-exploration as a
promising direction for RL exploration research.

本文对 Go-Explore 论文中未解决的问题进行了系统研究，通过对一系列 MiniGrid 环境的实验发现，采用后探索策略能够显著提高强化学习的性能，并利用新的自适应方法进一步提高了效果，证明了自适应后探索是强化学习探索研究的一个有前途的方向。