Addressing decision-making problems using sequence modeling to predict future
trajectories shows promising results in recent years. In this paper, we take a
step further to leverage the sequence predictive method in wider areas such as
long-term planning, vision-based control, and multi-task decision-making. To
this end, we propose a method to utilize a diffusion-based generative sequence
model to plan a series of milestones in a latent space and to have an agent to
follow the milestones to accomplish a given task. The proposed method can learn
control-relevant, low-dimensional latent representations of milestones, which
makes it possible to efficiently perform long-term planning and vision-based
control. Furthermore, our approach exploits generation flexibility of the
diffusion model, which makes it possible to plan diverse trajectories for
multi-task decision-making. We demonstrate the proposed method across offline
reinforcement learning (RL) benchmarks and an visual manipulation environment.
The results show that our approach outperforms offline RL methods in solving
long-horizon, sparse-reward tasks and multi-task problems, while also achieving
the state-of-the-art performance on the most challenging vision-based
manipulation benchmark.

利用序列建模方法来预测未来轨迹的决策问题在近年来取得了良好的结果。本文进一步利用这种序列预测方法在长期规划、基于视觉的控制和多任务决策等更广泛的领域中进行研究，提出了一种基于扩散模型的生成序列模型的方法，在潜空间内计划一系列里程碑，并使代理人遵循这些里程碑完成给定任务。该方法可以学习控制相关的低维潜变量表示，从而能够高效进行长期规划和基于视觉的控制。此外，我们的方法利用了扩散模型的生成灵活性，可以为多任务决策制定多样的轨迹。我们在离线强化学习（RL）基准测试和视觉操作环境中对该方法进行了验证，结果表明我们的方法在解决长远视角、稀疏奖励任务和多任务问题方面优于离线 RL 方法，并在最具挑战性的基于视觉的操作基准上实现了最先进的性能。