In the last few years, the research activity around reinforcement learning
tasks formulated over environments with sparse rewards has been especially
notable. Among the numerous approaches proposed to deal with these hard
exploration problems, intrinsic motivation mechanisms are arguably among the
most studied alternatives to date. Advances reported in this area over time
have tackled the exploration issue by proposing new algorithmic ideas to
generate alternative mechanisms to measure the novelty. However, most efforts
in this direction have overlooked the influence of different design choices and
parameter settings that have also been introduced to improve the effect of the
generated intrinsic bonus, forgetting the application of those choices to other
intrinsic motivation techniques that may also benefit of them. Furthermore,
some of those intrinsic methods are applied with different base reinforcement
algorithms (e.g. PPO, IMPALA) and neural network architectures, being hard to
fairly compare the provided results and the actual progress provided by each
solution. The goal of this work is to stress on this crucial matter in
reinforcement learning over hard exploration environments, exposing the
variability and susceptibility of avant-garde intrinsic motivation techniques
to diverse design factors. Ultimately, our experiments herein reported
underscore the importance of a careful selection of these design aspects
coupled with the exploration requirements of the environment and the task in
question under the same setup, so that fair comparisons can be guaranteed.

本文旨在探讨强化学习中的探索问题以及先锋内在动机技术对不同设计因素的可变性和易感性的影响，强调这些设计方面的谨慎选择以及环境和任务的探索要求，以保证公正比较。

应用内在动机技术于强化学习在难度较高的探索环境中的评估研究

An Evaluation Study of Intrinsic Motivation Techniques applied to Reinforcement Learning over Hard Exploration Environments

This paper introduces R2D3, an agent that makes efficient use of
demonstrations to solve hard exploration problems in partially observable
environments with highly variable initial conditions. We also introduce a suite
of eight tasks that combine these three properties, and show that R2D3 can
solve several of the tasks where other state of the art methods (both with and
without demonstrations) fail to see even a single successful trajectory after
tens of billions of steps of exploration.

介绍了 R2D3，这是一种代理程序，可有效利用演示来解决部分可观测环境下高度可变的难解探索问题。我们还介绍了一套结合这三个属性的八项任务，并展示了 R2D3 可以解决多项任务，在其他现有方法（无论是否使用演示）经过数十亿步探索后，仍无法看到一条成功路径。