Many real-world applications of reinforcement learning (RL) require the agent
to deal with high-dimensional observations such as those generated from a
megapixel camera. Prior work has addressed such problems with representation
learning, through which the agent can provably extract endogenous, latent state
information from raw observations and subsequently plan efficiently. However,
such approaches can fail in the presence of temporally correlated noise in the
observations, a phenomenon that is common in practice. We initiate the formal
study of latent state discovery in the presence of such exogenous noise sources
by proposing a new model, the Exogenous Block MDP (EX-BMDP), for rich
observation RL. We start by establishing several negative results, by
highlighting failure cases of prior representation learning based approaches.
Then, we introduce the Predictive Path Elimination (PPE) algorithm, that learns
a generalization of inverse dynamics and is provably sample and computationally
efficient in EX-BMDPs when the endogenous state dynamics are near
deterministic. The sample complexity of PPE depends polynomially on the size of
the latent endogenous state space while not directly depending on the size of
the observation space, nor the exogenous state space. We provide experiments on
challenging exploration problems which show that our approach works
empirically.

该论文研究了如何在处理高维观测数据时，通过引入先前的表示学习方法从原始观测数据中提取内生潜在状态信息，并在存在外生噪声时进行最优规划。提出了 EX-BMDP 模型，开展了潜在状态发现研究，并介绍了基于预测路径消除算法（PPE）的算法，对于近似确定性的内生状态动态具有很好的采样和计算效率。实验表明该方法具有良好的效果。