World models are progressively being employed across diverse fields,
extending from basic environment simulation to complex scenario construction.
However, existing models are mainly trained on domain-specific states and
actions, and confined to single-modality state representations. In this paper,
We introduce WorldGPT, a generalist world model built upon Multimodal Large
Language Model (MLLM). WorldGPT acquires an understanding of world dynamics
through analyzing millions of videos across various domains. To further enhance
WorldGPT's capability in specialized scenarios and long-term tasks, we have
integrated it with a novel cognitive architecture that combines memory
offloading, knowledge retrieval, and context reflection. As for evaluation, we
build WorldNet, a multimodal state transition prediction benchmark encompassing
varied real-life scenarios. Conducting evaluations on WorldNet directly
demonstrates WorldGPT's capability to accurately model state transition
patterns, affirming its effectiveness in understanding and predicting the
dynamics of complex scenarios. We further explore WorldGPT's emerging potential
in serving as a world simulator, helping multimodal agents generalize to
unfamiliar domains through efficiently synthesising multimodal instruction
instances which are proved to be as reliable as authentic data for fine-tuning
purposes. The project is available on
https://github.com/DCDmllm/WorldGPT.

构建于多模式大型语言模型（MLLM）之上的通用世界模型 WorldGPT，通过分析各领域的数百万个视频，使其理解世界动态。为了进一步增强 WorldGPT 在专门场景和长期任务中的能力，我们还将其与一种结合了内存卸载、知识检索和上下文反思的新型认知架构进行了集成。通过在涵盖各种现实情境的多模式状态转换预测基准 WorldNet 上进行评估，直接展示了 WorldGPT 准确建模状态转换模式的能力，确认其在理解和预测复杂情境动态方面的有效性。我们进一步探索了 WorldGPT 作为世界模拟器的潜力，通过高效合成多模式指令实例，帮助多模式代理在不熟悉的领域进行泛化，被证明与真实数据一样可靠用于微调目的。项目可在 https://github.com/DCDmllm/WorldGPT 找到。