Understanding the evolution of 3D scenes is important for effective
autonomous driving. While conventional methods mode scene development with the
motion of individual instances, world models emerge as a generative framework
to describe the general scene dynamics. However, most existing methods adopt an
autoregressive framework to perform next-token prediction, which suffer from
inefficiency in modeling long-term temporal evolutions. To address this, we
propose a diffusion-based 4D occupancy generation model, OccSora, to simulate
the development of the 3D world for autonomous driving. We employ a 4D scene
tokenizer to obtain compact discrete spatial-temporal representations for 4D
occupancy input and achieve high-quality reconstruction for long-sequence
occupancy videos. We then learn a diffusion transformer on the spatial-temporal
representations and generate 4D occupancy conditioned on a trajectory prompt.
We conduct extensive experiments on the widely used nuScenes dataset with Occ3D
occupancy annotations. OccSora can generate 16s-videos with authentic 3D layout
and temporal consistency, demonstrating its ability to understand the spatial
and temporal distributions of driving scenes. With trajectory-aware 4D
generation, OccSora has the potential to serve as a world simulator for the
decision-making of autonomous driving. Code is available at:
this https URL

理解 3D 场景的演变对于有效的自动驾驶至关重要。我们提出了一种基于扩散的 4D 占据生成模型 OccSora，以模拟自动驾驶中三维世界的发展。通过在空时表示中学习扩散变换器，我们可以在轨迹提示的条件下生成 4D 占据，展示了对驾驶场景的空间和时间分布的理解能力。具有轨迹感知的 4D 生成，OccSora 具有作为自动驾驶决策的世界模拟器的潜力。

OccSora: 自主驾驶中的 4D 占用生成模型作为世界模拟器

OccSora: 4D Occupancy Generation Models as World Simulators for  Autonomous Driving

World models are progressively being employed across diverse fields,
extending from basic environment simulation to complex scenario construction.
However, existing models are mainly trained on domain-specific states and
actions, and confined to single-modality state representations. In this paper,
We introduce WorldGPT, a generalist world model built upon Multimodal Large
Language Model (MLLM). WorldGPT acquires an understanding of world dynamics
through analyzing millions of videos across various domains. To further enhance
WorldGPT's capability in specialized scenarios and long-term tasks, we have
integrated it with a novel cognitive architecture that combines memory
offloading, knowledge retrieval, and context reflection. As for evaluation, we
build WorldNet, a multimodal state transition prediction benchmark encompassing
varied real-life scenarios. Conducting evaluations on WorldNet directly
demonstrates WorldGPT's capability to accurately model state transition
patterns, affirming its effectiveness in understanding and predicting the
dynamics of complex scenarios. We further explore WorldGPT's emerging potential
in serving as a world simulator, helping multimodal agents generalize to
unfamiliar domains through efficiently synthesising multimodal instruction
instances which are proved to be as reliable as authentic data for fine-tuning
purposes. The project is available on
https://github.com/DCDmllm/WorldGPT.

构建于多模式大型语言模型（MLLM）之上的通用世界模型 WorldGPT，通过分析各领域的数百万个视频，使其理解世界动态。为了进一步增强 WorldGPT 在专门场景和长期任务中的能力，我们还将其与一种结合了内存卸载、知识检索和上下文反思的新型认知架构进行了集成。通过在涵盖各种现实情境的多模式状态转换预测基准 WorldNet 上进行评估，直接展示了 WorldGPT 准确建模状态转换模式的能力，确认其在理解和预测复杂情境动态方面的有效性。我们进一步探索了 WorldGPT 作为世界模拟器的潜力，通过高效合成多模式指令实例，帮助多模式代理在不熟悉的领域进行泛化，被证明与真实数据一样可靠用于微调目的。项目可在 https://github.com/DCDmllm/WorldGPT 找到。