It has been established that diverse behaviors spanning the controllable
subspace of an Markov decision process can be trained by rewarding a policy for
being distinguishable from other policies \citep{gregor2016variational,
eysenbach2018diversity, warde2018unsupervised}. However, one limitation of this
formulation is generalizing behaviors beyond the finite set being explicitly
learned, as is needed for use on subsequent tasks. Successor features
\citep{dayan93improving, barreto2017successor} provide an appealing solution to
this generalization problem, but require defining the reward function as linear
in some grounded feature space. In this paper, we show that these two
techniques can be combined, and that each method solves the other's primary
limitation. To do so we introduce Variational Intrinsic Successor FeatuRes
(VISR), a novel algorithm which learns controllable features that can be
leveraged to provide enhanced generalization and fast task inference through
the successor feature framework. We empirically validate VISR on the full Atari
suite, in a novel setup wherein the rewards are only exposed briefly after a
long unsupervised phase. Achieving human-level performance on 14 games and
beating all baselines, we believe VISR represents a step towards agents that
rapidly learn from limited feedback.

本文介绍了一种将马尔可夫决策过程和继承特征相结合的算法 (VISR)，该算法通过学习可控特征对继承特征进行增强，从而实现对 Atari 游戏中奖励的快速学习和泛化能力，并在 14 个游戏中实现了人类水平的表现。