Intelligent agents can cope with sensory-rich environments by learning task-agnostic state abstractions. In this paper, we propose mechanisms to approximate causal states, which optimally compress the joint history of actions and observations in partially-observable Markov decision processes. Our proposed algorithm extracts causal state representations from RNNs that are trained to predict subsequent observations given the history. We demonstrate that these learned task-agnostic state abstractions can be used to efficiently learn policies for reinforcement learning problems with rich observation spaces. We evaluate agents using multiple partially observable navigation tasks with both discrete (GridWorld) and continuous (VizDoom, ALE) observation processes that cannot be solved by traditional memory-limited methods. Our experiments demonstrate systematic improvement of the DQN and tabular models using approximate causal state representations with respect to recurrent-DQN baselines trained with raw inputs.

本文提出了一种基于循环神经网络（RNN）的近似因果状态算法，该方法学习从POMDP中的历史动作和观察预测未来观察情况的因果状态表示。实验证明，所学习的状态表示可用于有效学习具有丰富观察空间的强化学习问题，并与之前的方法进行比较。

学习部分可观测环境的因果状态表示