In model-based reinforcement learning, the agent interleaves between model
learning and planning. These two components are inextricably intertwined. If
the model is not able to provide sensible long-term prediction, the executed
planner would exploit model flaws, which can yield catastrophic failures. This
paper focuses on building a model that reasons about the long-term future and
demonstrates how to use this for efficient planning and exploration. To this
end, we build a latent-variable autoregressive model by leveraging recent ideas
in variational inference. We argue that forcing latent variables to carry
future information through an auxiliary task substantially improves long-term
predictions. Moreover, by planning in the latent space, the planner's solution
is ensured to be within regions where the model is valid. An exploration
strategy can be devised by searching for unlikely trajectories under the model.
Our method achieves higher reward faster compared to baselines on a variety of
tasks and environments in both the imitation learning and model-based
reinforcement learning settings.

本文着重于构建一个具有考虑长期未来的模型，并展示如何利用它进行有效规划和探索，通过搜寻模型下的不可能轨迹来设计探索策略，并在两种学习环境中取得了比基线更快更高报酬的效果。