Automatically generating high-quality real world 3D scenes is of enormous
interest for applications such as virtual reality and robotics simulation.
Towards this goal, we introduce NeuralField-LDM, a generative model capable of
synthesizing complex 3D environments. We leverage Latent Diffusion Models that
have been successfully utilized for efficient high-quality 2D content creation.
We first train a scene auto-encoder to express a set of image and pose pairs as
a neural field, represented as density and feature voxel grids that can be
projected to produce novel views of the scene. To further compress this
representation, we train a latent-autoencoder that maps the voxel grids to a
set of latent representations. A hierarchical diffusion model is then fit to
the latents to complete the scene generation pipeline. We achieve a substantial
improvement over existing state-of-the-art scene generation models.
Additionally, we show how NeuralField-LDM can be used for a variety of 3D
content creation applications, including conditional scene generation, scene
inpainting and scene style manipulation.

提出了一种名为 NeuralField-LDM 的生成模型，它能够合成复杂的 3D 环境，并取得了比目前现有模型更显著的进步。该模型使用自动编码器训练场景图片，通过特征体素格点表示神经场，并进一步利用潜在自动编码器压缩、映射体素格点至潜在空间，并采用分层扩散模型完成 3D 场景生成管线。此外，展示了如何应用 NeuralField-LDM 在条件场景生成、场景修复和场景风格变化等 3D 内容创作应用领域。