We address the challenge of recovering an underlying scene geometry and colors from a sparse set of RGBD view observations. In this work, we present a new solution that sequentially generates novel RGBD views along a camera trajectory, and the scene geometry is simply the fusion result of these views. More specifically, we maintain an intermediate surface mesh used for rendering new RGBD views, which subsequently becomes complete by an inpainting network; each rendered RGBD view is later back-projected as a partial surface and is supplemented into the intermediate mesh. The use of intermediate mesh and camera projection helps solve the refractory problem of multi-view inconsistency. We practically implement the RGBD inpainting network as a versatile RGBD diffusion model, which is previously used for 2D generative modeling; we make a modification to its reverse diffusion process to enable our use. We evaluate our approach on the task of 3D scene synthesis from sparse RGBD inputs; extensive experiments on the ScanNet dataset demonstrate the superiority of our approach over existing ones. Project page: https://jblei.site/project-pages/rgbd-diffusion.html

该研究提出了一个新的解决方案 - RGBD^2，它可以从稀疏的RGBD视图中顺序生成新的RGBD视图，并将场景几何形状作为这些视图的融合结果。通过维护中间表面网格，并使用摄像机投影，将渲染的RGBD视图再次投影为部分表面并补充到中间网格中，以解决多视角不一致性的问题。这种方法在稀疏的RGBD输入中合成3D场景时具有优越性。

RGBD2：使用RGBD扩散模型进行增量式视图修复的生成场景合成