We introduce SceneDiffuser, a conditional generative model for 3D scene
understanding. SceneDiffuser provides a unified model for solving
scene-conditioned generation, optimization, and planning. In contrast to prior
works, SceneDiffuser is intrinsically scene-aware, physics-based, and
goal-oriented. With an iterative sampling strategy, SceneDiffuser jointly
formulates the scene-aware generation, physics-based optimization, and
goal-oriented planning via a diffusion-based denoising process in a fully
differentiable fashion. Such a design alleviates the discrepancies among
different modules and the posterior collapse of previous scene-conditioned
generative models. We evaluate SceneDiffuser with various 3D scene
understanding tasks, including human pose and motion generation, dexterous
grasp generation, path planning for 3D navigation, and motion planning for
robot arms. The results show significant improvements compared with previous
models, demonstrating the tremendous potential of SceneDiffuser for the broad
community of 3D scene understanding.

SceneDiffuser 是一个用于 3D 场景理解的条件生成模型，通过扩散过程，联合制定了场景感知生成、基于物理的优化和面向目标的规划模块，相对于之前的模型，具有内在的场景感知、基于物理的设计器和面向目标的设计等优点，在人体姿态和运动生成、灵巧握持生成、三维导航路径规划和机器人臂运动规划等任务上表现了极大的潜力。