The ability to plan into the future while utilizing only raw high-dimensional
observations, such as images, can provide autonomous agents with broad
capabilities. Visual model-based reinforcement learning (RL) methods that plan
future actions directly have shown impressive results on tasks that require
only short-horizon reasoning, however, these methods struggle on temporally
extended tasks. We argue that it is easier to solve long-horizon tasks by
planning sequences of states rather than just actions, as the effects of
actions greatly compound over time and are harder to optimize. To achieve this,
we draw on the idea of collocation, which has shown good results on
long-horizon tasks in optimal control literature, and adapt it to the
image-based setting by utilizing learned latent state space models. The
resulting latent collocation method (LatCo) optimizes trajectories of latent
states, which improves over previously proposed shooting methods for visual
model-based RL on tasks with sparse rewards and long-term goals. Videos and
code at this https URL

该研究使用学习的潜在状态空间模型，结合目测模型强化学习方法，提出了一种优化潜在状态轨迹的 LatCo 方法，来解决长时程、奖励稀疏的任务。通过序列规划，相比于之前使用的射击方法，该方法在历经时间过程中的效果更好。