While controllable generative models for images and videos have achieved
remarkable success, high-quality models for 3D scenes, particularly in
unbounded scenarios like autonomous driving, remain underdeveloped due to high
data acquisition costs. In this paper, we introduce MagicDrive3D, a novel
pipeline for controllable 3D street scene generation that supports
multi-condition control, including BEV maps, 3D objects, and text descriptions.
Unlike previous methods that reconstruct before training the generative models,
MagicDrive3D first trains a video generation model and then reconstructs from
the generated data. This innovative approach enables easily controllable
generation and static scene acquisition, resulting in high-quality scene
reconstruction. To address the minor errors in generated content, we propose
deformable Gaussian splatting with monocular depth initialization and
appearance modeling to manage exposure discrepancies across viewpoints.
Validated on the nuScenes dataset, MagicDrive3D generates diverse, high-quality
3D driving scenes that support any-view rendering and enhance downstream tasks
like BEV segmentation. Our results demonstrate the framework's superior
performance, showcasing its transformative potential for autonomous driving
simulation and beyond.

我们引入了 MagicDrive3D，这是一种创新的管道，用于可控的 3D 街道场景生成，支持多条件控制，包括 BEV 地图、3D 对象和文本描述。与先前的方法不同的是，MagicDrive3D 首先训练视频生成模型，然后再进行重建，从而实现了易于控制的生成和静态场景获取，从而实现了高质量的场景重建。通过对 nuScenes 数据集的验证，MagicDrive3D 生成了多样化、高质量的 3D 驾驶场景，支持任意视角渲染，并增强了诸如 BEV 分割之类的下游任务。我们的结果展示了该框架卓越的性能，展示了它在自动驾驶仿真及其他领域的变革潜力。