We are interested in enabling visual planning for complex long-horizon tasks
in the space of generated videos and language, leveraging recent advances in
large generative models pretrained on Internet-scale data. To this end, we
present video language planning (VLP), an algorithm that consists of a tree
search procedure, where we train (i) vision-language models to serve as both
policies and value functions, and (ii) text-to-video models as dynamics models.
VLP takes as input a long-horizon task instruction and current image
observation, and outputs a long video plan that provides detailed multimodal
(video and language) specifications that describe how to complete the final
task. VLP scales with increasing computation budget where more computation time
results in improved video plans, and is able to synthesize long-horizon video
plans across different robotics domains: from multi-object rearrangement, to
multi-camera bi-arm dexterous manipulation. Generated video plans can be
translated into real robot actions via goal-conditioned policies, conditioned
on each intermediate frame of the generated video. Experiments show that VLP
substantially improves long-horizon task success rates compared to prior
methods on both simulated and real robots (across 3 hardware platforms).

我们介绍了视频语言规划（VLP）算法，利用在互联网规模数据上预训练的大型生成模型，通过在生成视频和语言空间中进行可视化规划来完成复杂的长期任务，通过针对多模态（视频和语言）进行详细规划来描述如何完成最终任务，进一步通过目标条件策略将生成的视频转化为真实的机器人行动。