The success of Reinforcement Learning (RL) heavily relies on the ability to
learn robust representations from the observations of the environment. In most
cases, the representations learned purely by the reinforcement learning loss
can differ vastly across states depending on how the value functions change.
However, the representations learned need not be very specific to the task at
hand. Relying only on the RL objective may yield representations that vary
greatly across successive time steps. In addition, since the RL loss has a
changing target, the representations learned would depend on how good the
current values/policies are. Thus, disentangling the representations from the
main task would allow them to focus more on capturing transition dynamics which
can improve generalization. To this end, we propose locally constrained
representations, where an auxiliary loss forces the state representations to be
predictable by the representations of the neighbouring states. This encourages
the representations to be driven not only by the value/policy learning but also
self-supervised learning, which constrains the representations from changing
too rapidly. We evaluate the proposed method on several known benchmarks and
observe strong performance. Especially in continuous control tasks, our
experiments show a significant advantage over a strong baseline.

本文提出一种局部约束表示法，通过对环境观测状态的预测及邻近状态的表示作为辅助损失，将强化学习中的表示与任务相分离，可以提高泛化能力，有效应用于连续控制任务中。