We present "SemCity," a 3D diffusion model for semantic scene generation in
real-world outdoor environments. Most 3D diffusion models focus on generating a
single object, synthetic indoor scenes, or synthetic outdoor scenes, while the
generation of real-world outdoor scenes is rarely addressed. In this paper, we
concentrate on generating a real-outdoor scene through learning a diffusion
model on a real-world outdoor dataset. In contrast to synthetic data,
real-outdoor datasets often contain more empty spaces due to sensor
limitations, causing challenges in learning real-outdoor distributions. To
address this issue, we exploit a triplane representation as a proxy form of
scene distributions to be learned by our diffusion model. Furthermore, we
propose a triplane manipulation that integrates seamlessly with our triplane
diffusion model. The manipulation improves our diffusion model's applicability
in a variety of downstream tasks related to outdoor scene generation such as
scene inpainting, scene outpainting, and semantic scene completion refinements.
In experimental results, we demonstrate that our triplane diffusion model shows
meaningful generation results compared with existing work in a real-outdoor
dataset, SemanticKITTI. We also show our triplane manipulation facilitates
seamlessly adding, removing, or modifying objects within a scene. Further, it
also enables the expansion of scenes toward a city-level scale. Finally, we
evaluate our method on semantic scene completion refinements where our
diffusion model enhances predictions of semantic scene completion networks by
learning scene distribution. Our code is available at
this https URL

我们提出了一种用于在现实世界的室外环境中生成语义场景的 3D 扩散模型 “SemCity”。通过在现实世界的室外数据集上学习扩散模型，我们集中于生成真实室外场景。我们利用三面板表示作为我们扩散模型要学习的场景分布的代理形式，并提出了与我们的三面板扩散模型无缝集成的三面板操作。我们的实验结果表明，与现有的工作相比，我们的三面板扩散模型在真实室外数据集 SemanticKITTI 中展示了有意义的生成结果。同时，我们的三面板操作使得能够在场景中添加、删除或修改对象，并且还能将场景扩展到城市级别。最后，我们通过学习场景分布，评估我们的方法在语义场景补全的完善上的作用。

SemCity: 带三层扩散的语义场景生成

SemCity: Semantic Scene Generation with Triplane Diffusion

Generating accurate 3D models is a challenging problem that traditionally
requires explicit learning from 3D datasets using supervised learning. Although
recent advances have shown promise in learning 3D models from 2D images, these
methods often rely on well-structured datasets with multi-view images of each
instance or camera pose information. Furthermore, these datasets usually
contain clean backgrounds with simple shapes, making them expensive to acquire
and hard to generalize, which limits the applicability of these methods. To
overcome these limitations, we propose a method for reconstructing 3D geometry
from the diverse and unstructured Imagenet dataset without camera pose
information. We use an efficient triplane representation to learn 3D models
from 2D images and modify the architecture of the generator backbone based on
StyleGAN2 to adapt to the highly diverse dataset. To prevent mode collapse and
improve the training stability on diverse data, we propose to use multi-view
discrimination. The trained generator can produce class-conditional 3D models
as well as renderings from arbitrary viewpoints. The class-conditional
generation results demonstrate significant improvement over the current
state-of-the-art method. Additionally, using PTI, we can efficiently
reconstruct the whole 3D geometry from single-view images.

从多样且非结构化的 Imagenet 数据集中，我们提出了一种从二维图像中重建三维几何模型的方法，使用高效的三平面表示法学习三维模型，并基于 StyleGAN2 的生成器架构对高度多样化的数据集进行调整，通过多视点辨别防止模式崩溃并提升训练稳定性。