Representations are at the core of all deep reinforcement learning (RL)
methods for both Markov decision processes (MDPs) and partially observable
Markov decision processes (POMDPs). Many representation learning methods and
theoretical frameworks have been developed to understand what constitutes an
effective representation. However, the relationships between these methods and
the shared properties among them remain unclear. In this paper, we show that
many of these seemingly distinct methods and frameworks for state and history
abstractions are, in fact, based on a common idea of self-predictive
abstraction. Furthermore, we provide theoretical insights into the widely
adopted objectives and optimization, such as the stop-gradient technique, in
learning self-predictive representations. These findings together yield a
minimalist algorithm to learn self-predictive representations for states and
histories. We validate our theories by applying our algorithm to standard MDPs,
MDPs with distractors, and POMDPs with sparse rewards. These findings culminate
in a set of practical guidelines for RL practitioners.

深度强化学习的关键是表示方法，这篇论文揭示了多种表示学习方法和理论框架之间的共同性，特别是基于自预测抽象的思想，并给出了学习自预测表示方法的最简算法和实用指南。