The focus of this work is sample-efficient deep reinforcement learning (RL) with a simulator. One useful property of simulators is that it is typically easy to reset the environment to a previously observed state. We propose an algorithmic framework, named uncertainty-first local planning (UFLP), that takes advantage of this property. Concretely, in each data collection iteration, with some probability, our meta-algorithm resets the environment to an observed state which has high uncertainty, instead of sampling according to the initial-state distribution. The agent-environment interaction then proceeds as in the standard online RL setting. We demonstrate that this simple procedure can dramatically improve the sample cost of several baseline RL algorithms on difficult exploration tasks. Notably, with our framework, we can achieve super-human performance on the notoriously hard Atari game, Montezuma's Revenge, with a simple (distributional) double DQN. Our work can be seen as an efficient approximate implementation of an existing algorithm with theoretical guarantees, which offers an interpretation of the positive empirical results.

本文提出了一种名为“不确定性优先本地规划”的算法框架，结合模拟器的属性，在每一次数据收集迭代中，以一定概率将环境重置到高度不确定性的已观测状态，这样可以显著提高几个基准强化学习算法在困难的探索任务上的样本成本，并在 Atari 游戏Montezuma's Revenge中实现了超人类性能。

通过本地规划实现样本高效深度强化学习