Task planning for embodied AI has been one of the most challenging problems
where the community does not meet a consensus in terms of formulation. In this
paper, we aim to tackle this problem with a unified framework consisting of an
end-to-end trainable method and a planning algorithm. Particularly, we propose
a task-agnostic method named 'planning as in-painting'. In this method, we use
a Denoising Diffusion Model (DDM) for plan generation, conditioned on both
language instructions and perceptual inputs under partially observable
environments. Partial observation often leads to the model hallucinating the
planning. Therefore, our diffusion-based method jointly models both state
trajectory and goal estimation to improve the reliability of the generated
plan, given the limited available information at each step. To better leverage
newly discovered information along the plan execution for a higher success
rate, we propose an on-the-fly planning algorithm to collaborate with the
diffusion-based planner. The proposed framework achieves promising performances
in various embodied AI tasks, including vision-language navigation, object
manipulation, and task planning in a photorealistic virtual environment. The
code is available at: this https URL

在这篇研究论文中，我们提出了一个统一的框架，包括端到端可训练方法和规划算法，用于解决具有挑战性的具体任务规划问题。我们使用一种名为 'planning as in-painting' 的任务无关方法，在部分可观察环境下，结合语言指令和感知输入条件下的去噪扩散模型（DDM）来生成计划。为了提高生成计划的可靠性，我们的扩散方法同时建模状态轨迹和目标估计，并提出了一种即时规划算法与扩散规划器协作。该框架在包括视觉语言导航、物体操作和任务规划在真实虚拟环境中的表现表现出有希望的性能。