Learning task-relevant state representations is crucial to solving the
problem of scene generalization in visual deep reinforcement learning. Prior
work typically establishes a self-supervised auxiliary learner, introducing
elements (e.g., rewards and actions) to extract task-relevant state information
from observations through behavioral similarity metrics. However, the methods
often ignore the inherent relationships between the elements (e.g., dynamics
relationships) that are essential for learning accurate representations, and
they are also limited to single-step metrics, which impedes the discrimination
of short-term similar task/behavior information in long-term dynamics
transitions. To solve the issues, we propose an intrinsic dynamic
characteristics-driven sequence representation learning method (DSR) over a
common DRL frame. Concretely, inspired by the fact of state transition in the
underlying system, it constrains the optimization of the encoder via modeling
the dynamics equations related to the state transition, which prompts the
latent encoding information to satisfy the state transition process and thereby
distinguishes state space and noise space. Further, to refine the ability of
encoding similar tasks based on dynamics constraints, DSR also sequentially
models inherent dynamics equation relationships from the perspective of
sequence elements' frequency domain and multi-step prediction. Finally,
experimental results show that DSR has achieved a significant performance boost
in the Distracting DMControl Benchmark, with an average of 78.9% over the
backbone baseline. Further results indicate that it also achieves the best
performance in real-world autonomous driving tasks in the CARLA simulator.
Moreover, the qualitative analysis results of t-SNE visualization validate that
our method possesses superior representation ability on visual tasks.

通过建模状态转换的动力学方程、优化编码器以满足状态转换过程并区分状态空间和噪声空间，以及从序列元素频率域和多步预测的角度顺序地建模内在动力学方程关系，我们提出了一种内在动力学特征驱动的序列表示学习方法（DSR），并在视觉任务中展示了出色的表征能力。