Selecting exploratory actions that generate a rich stream of experience for
better learning is a fundamental challenge in reinforcement learning (RL). An
approach to tackle this problem consists in selecting actions according to
specific policies for an extended period of time, also known as options. A
recent line of work to derive such exploratory options builds upon the
eigenfunctions of the graph Laplacian. Importantly, until now these methods
have been mostly limited to tabular domains where (1) the graph Laplacian
matrix was either given or could be fully estimated, (2) performing
eigendecomposition on this matrix was computationally tractable, and (3) value
functions could be learned exactly. Additionally, these methods required a
separate option discovery phase. These assumptions are fundamentally not
scalable. In this paper we address these limitations and show how recent
results for directly approximating the eigenfunctions of the Laplacian can be
leveraged to truly scale up options-based exploration. To do so, we introduce a
fully online deep RL algorithm for discovering Laplacian-based options and
evaluate our approach on a variety of pixel-based tasks. We compare to several
state-of-the-art exploration methods and show that our approach is effective,
general, and especially promising in non-stationary settings.

本文提出一种基于图拉普拉斯算子的在线深度强化学习算法，用于生成探索选项，通过在基于像素的任务中的比较，证明了该算法的有效性、通用性和非静态环境下的优越性。