Controllable synthetic data generation can substantially lower the annotation
cost of training data in autonomous driving research and development. Prior
works use diffusion models to generate driving images conditioned on the 3D
object layout. However, those models are trained on small-scale datasets like
nuScenes, which lack appearance and layout diversity. Moreover, the trained
models can only generate images based on the real-world layout data from the
validation set of the same dataset, where overfitting might happen. In this
work, we introduce a simulator-conditioned scene generation framework called
SimGen that can learn to generate diverse driving scenes by mixing data from
the simulator and the real world. It uses a novel cascade diffusion pipeline to
address challenging sim-to-real gaps and multi-condition conflicts. A driving
video dataset DIVA is collected to enhance the generative diversity of SimGen,
which contains over 147.5 hours of real-world driving videos from 73 locations
worldwide and simulated driving data from the MetaDrive simulator. SimGen
achieves superior generation quality and diversity while preserving
controllability based on the text prompt and the layout pulled from a
simulator. We further demonstrate the improvements brought by SimGen for
synthetic data augmentation on the BEV detection and segmentation task and
showcase its capability in safety-critical data generation. Code, data, and
models will be made available.

通过混合模拟器和现实世界的数据，SimGen 模型能够学习生成具有多样性的驾驶场景，通过新颖的级联扩散管道解决了模拟到真实世界之间的差距和多条件冲突，并在保持可控性的基础上，获得卓越的生成质量和多样性。