Learning models of the environment from pure interaction is often considered
an essential component of building lifelong reinforcement learning agents.
However, the common practice in model-based reinforcement learning is to learn
models that model every aspect of the agent's environment, regardless of
whether they are important in coming up with optimal decisions or not. In this
paper, we argue that such models are not particularly well-suited for
performing scalable and robust planning in lifelong reinforcement learning
scenarios and we propose new kinds of models that only model the relevant
aspects of the environment, which we call "minimal value-equivalent partial
models". After providing a formal definition for these models, we provide
theoretical results demonstrating the scalability advantages of performing
planning with such models and then perform experiments to empirically
illustrate our theoretical results. Then, we provide some useful heuristics on
how to learn these kinds of models with deep learning architectures and
empirically demonstrate that models learned in such a way can allow for
performing planning that is robust to distribution shifts and compounding model
errors. Overall, both our theoretical and empirical results suggest that
minimal value-equivalent partial models can provide significant benefits to
performing scalable and robust planning in lifelong reinforcement learning
scenarios.

本文提出了新的模型，称为 “最小价值等效部分模型”，它只对环境的相关方面进行建模。通过实验证明，基于此类模型的规划方法具有可伸缩性和鲁棒性。