Generating unbounded 3D scenes is crucial for large-scale scene understanding
and simulation. Urban scenes, unlike natural landscapes, consist of various
complex man-made objects and structures such as roads, traffic signs, vehicles,
and buildings. To create a realistic and detailed urban scene, it is crucial to
accurately represent the geometry and semantics of the underlying objects,
going beyond their visual appearance. In this work, we propose UrbanDiffusion,
a 3D diffusion model that is conditioned on a Bird's-Eye View (BEV) map and
generates an urban scene with geometry and semantics in the form of semantic
occupancy map. Our model introduces a novel paradigm that learns the data
distribution of scene-level structures within a latent space and further
enables the expansion of the synthesized scene into an arbitrary scale. After
training on real-world driving datasets, our model can generate a wide range of
diverse urban scenes given the BEV maps from the held-out set and also
generalize to the synthesized maps from a driving simulator. We further
demonstrate its application to scene image synthesis with a pretrained image
generator as a prior.

利用 Bird's-Eye View 地图条件的 3D 扩散模型 UrbanDiffusion 生成具有几何和语义的城市场景，并通过训练实际驾驶数据集，该模型能够生成多样性的城市场景，进一步将生成的场景扩展到任意规模，并通过预训练的图像生成器实现场景图像合成的应用。