We introduce a new unsupervised pretraining objective for reinforcement
learning. During the unsupervised reward-free pretraining phase, the agent
maximizes mutual information between tasks and states induced by the policy.
Our key contribution is a novel lower bound of this intractable quantity. We
show that by reinterpreting and combining variational successor
features~\citep{Hansen2020Fast} with nonparametric entropy
maximization~\citep{liu2021behavior}, the intractable mutual information can be
efficiently optimized. The proposed method Active Pretraining with Successor
Feature (APS) explores the environment via nonparametric entropy maximization,
and the explored data can be efficiently leveraged to learn behavior by
variational successor features. APS addresses the limitations of existing
mutual information maximization based and entropy maximization based
unsupervised RL, and combines the best of both worlds. When evaluated on the
Atari 100k data-efficiency benchmark, our approach significantly outperforms
previous methods combining unsupervised pretraining with task-specific
finetuning.

本文提出了一种新的强化学习的无监督预训练目标，利用非参数熵最大化和变分继承者特征相结合，提出了 APS 方法，比现有方法表现更好。APS 通过非参数熵最大化探索环境数据，通过变分继承者特征学习行为。在 Atari 100k 数据效率基准测试中，APS 方法比结合无监督预训练和任务特定调参的先前方法表现显著更好。