We aim to tackle sparse-view reconstruction of a 360 3D scene using priors from latent diffusion models (LDM). The sparse-view setting is ill-posed and underconstrained, especially for scenes where the camera rotates 360 degrees around a point, as no visual information is available beyond some frontal views focused on the central object(s) of interest. In this work, we show that pretrained 2D diffusion models can strongly improve the reconstruction of a scene with low-cost fine-tuning. Specifically, we present SparseSplat360 (Sp2360), a method that employs a cascade of in-painting and artifact removal models to fill in missing details and clean novel views. Due to superior training and rendering speeds, we use an explicit scene representation in the form of 3D Gaussians over NeRF-based implicit representations. We propose an iterative update strategy to fuse generated pseudo novel views with existing 3D Gaussians fitted to the initial sparse inputs. As a result, we obtain a multi-view consistent scene representation with details coherent with the observed inputs. Our evaluation on the challenging Mip-NeRF360 dataset shows that our proposed 2D to 3D distillation algorithm considerably improves the performance of a regularized version of 3DGS adapted to a sparse-view setting and outperforms existing sparse-view reconstruction methods in 360 scene reconstruction. Qualitatively, our method generates entire 360 scenes from as few as 9 input views, with a high degree of foreground and background detail.

通过使用潜在扩散模型（LDM）的先验来解决360度3D场景的稀疏视图重建问题。我们展示了通过预训练的2D扩散模型经过精细调整可以大幅提高场景重建的方法SparseSplat360（Sp2360），该方法通过级联修补和伪影移除模型填补缺失的细节并清除新的视图。我们提出了一种迭代更新策略，将生成的伪新视图与拟合到初始稀疏输入的3D高斯模型进行融合，从而获得与观察输入一致且具有上下文的细节的多视图场景表示。在具有挑战性的Mip-NeRF360数据集上的评估结果表明，我们提出的2D到3D蒸馏算法显著改善了用于稀疏视图的正则化版本的3DGS的性能，并且在360度场景重建方面优于现有的稀疏视图重建方法。定性上，我们的方法可以从仅有9个输入视图生成具有前景和背景细节的完整360度场景。

Sp2360：使用级联2D扩散先验进行稀疏角度360场景重建