World models, especially in autonomous driving, are trending and drawing
extensive attention due to their capacity for comprehending driving
environments. The established world model holds immense potential for the
generation of high-quality driving videos, and driving policies for safe
maneuvering. However, a critical limitation in relevant research lies in its
predominant focus on gaming environments or simulated settings, thereby lacking
the representation of real-world driving scenarios. Therefore, we introduce
DriveDreamer, a pioneering world model entirely derived from real-world driving
scenarios. Regarding that modeling the world in intricate driving scenes
entails an overwhelming search space, we propose harnessing the powerful
diffusion model to construct a comprehensive representation of the complex
environment. Furthermore, we introduce a two-stage training pipeline. In the
initial phase, DriveDreamer acquires a deep understanding of structured traffic
constraints, while the subsequent stage equips it with the ability to
anticipate future states. The proposed DriveDreamer is the first world model
established from real-world driving scenarios. We instantiate DriveDreamer on
the challenging nuScenes benchmark, and extensive experiments verify that
DriveDreamer empowers precise, controllable video generation that faithfully
captures the structural constraints of real-world traffic scenarios.
Additionally, DriveDreamer enables the generation of realistic and reasonable
driving policies, opening avenues for interaction and practical applications.

DriveDreamer 是第一个从真实世界驾驶场景中建立的世界模型，通过使用扩散模型和两阶段训练流程，能够准确地生成高质量驾驶视频，并捕捉到真实世界交通场景的结构约束。

DriveDreamer：面向实际驱动的自动驾驶世界模型

DriveDreamer: Towards Real-world-driven World Models for Autonomous  Driving

Fully autonomous driving has been widely studied and is becoming increasingly
feasible. However, such autonomous driving has yet to be achieved on public
roads, because of various uncertainties due to surrounding human drivers and
pedestrians. In this paper, we present an end-to-end learningbased autonomous
driving system named SuperDriver AI, where Deep Neural Networks (DNNs) learn
the driving actions and policies from the experienced human drivers and
determine the driving maneuvers to take while guaranteeing road safety. In
addition, to improve robustness and interpretability, we present a slit model
and a visual attention module. We build a datacollection system and emulator
with real-world hardware, and we also test the SuperDriver AI system with
real-world driving scenarios. Finally, we have collected 150 runs for one
driving scenario in Tokyo, Japan, and have shown the demonstration of
SuperDriver AI with the real-world vehicle.

本研究提出基于深度神经网络的端到端学习自动驾驶系统 SuperDriver AI，该系统学习来自经验丰富的人类驾驶员的驾驶行为和政策，采取驾驶操作并确保路面安全。实验结果表明，该系统在真实行驶场景下运行良好。