Automatically generating high-quality real world 3D scenes is of enormous interest for applications such as virtual reality and robotics simulation. Towards this goal, we introduce NeuralField-LDM, a generative model capable of synthesizing complex 3D environments. We leverage Latent Diffusion Models that have been successfully utilized for efficient high-quality 2D content creation. We first train a scene auto-encoder to express a set of image and pose pairs as a neural field, represented as density and feature voxel grids that can be projected to produce novel views of the scene. To further compress this representation, we train a latent-autoencoder that maps the voxel grids to a set of latent representations. A hierarchical diffusion model is then fit to the latents to complete the scene generation pipeline. We achieve a substantial improvement over existing state-of-the-art scene generation models. Additionally, we show how NeuralField-LDM can be used for a variety of 3D content creation applications, including conditional scene generation, scene inpainting and scene style manipulation.

提出了一种名为 NeuralField-LDM 的生成模型，它能够合成复杂的 3D 环境，并取得了比目前现有模型更显著的进步。该模型使用自动编码器训练场景图片，通过特征体素格点表示神经场，并进一步利用潜在自动编码器压缩、映射体素格点至潜在空间，并采用分层扩散模型完成 3D 场景生成管线。此外，展示了如何应用 NeuralField-LDM 在条件场景生成、场景修复和场景风格变化等 3D 内容创作应用领域。

神经场 LDM: 带有分层隐式扩散模型的场景生成