World models empower model-based agents to interactively explore, reason, and
plan within imagined environments for real-world decision-making. However, the
high demand for interactivity poses challenges in harnessing recent
advancements in video generative models for developing world models at scale.
This work introduces Interactive VideoGPT (iVideoGPT), a scalable
autoregressive transformer framework that integrates multimodal signals--visual
observations, actions, and rewards--into a sequence of tokens, facilitating an
interactive experience of agents via next-token prediction. iVideoGPT features
a novel compressive tokenization technique that efficiently discretizes
high-dimensional visual observations. Leveraging its scalable architecture, we
are able to pre-train iVideoGPT on millions of human and robotic manipulation
trajectories, establishing a versatile foundation that is adaptable to serve as
interactive world models for a wide range of downstream tasks. These include
action-conditioned video prediction, visual planning, and model-based
reinforcement learning, where iVideoGPT achieves competitive performance
compared with state-of-the-art methods. Our work advances the development of
interactive general world models, bridging the gap between generative video
models and practical model-based reinforcement learning applications.

通过引入 Interactive VideoGPT (iVideoGPT) 框架，可以在实现世界决策过程中，使模型交互地探索、推理和规划。 iVideoGPT 通过将多模态信号（视觉观察，行为和奖励）整合成一系列令牌，实现了通过下一个令牌预测的交互式体验，其中包括一种高效离散化高维视觉观察的新型压缩词汇化技术。借助其可扩展的架构，我们能够在数百万人类和机器人操作轨迹上进行预训练，建立起适用于各种下游任务的多功能基础，其中包括动作条件的视频预测，可视化规划和基于模型的强化学习，并且在与最先进方法相比，iVideoGPT 实现了竞争性的性能。我们的工作推进了交互式通用世界模型的发展，弥合了生成式视频模型和实际模型导向强化学习应用之间的差距。

iVideoGPT: 可扩展世界模型的交互式 VideoGPT

iVideoGPT: Interactive VideoGPTs are Scalable World Models

Recently, model-based agents have achieved better performance than model-free
ones using the same computational budget and training time in single-agent
environments. However, due to the complexity of multi-agent systems, it is
tough to learn the model of the environment. The significant compounding error
may hinder the learning process when model-based methods are applied to
multi-agent tasks. This paper proposes an implicit model-based multi-agent
reinforcement learning method based on value decomposition methods. Under this
method, agents can interact with the learned virtual environment and evaluate
the current state value according to imagined future states in the latent
space, making agents have the foresight. Our approach can be applied to any
multi-agent value decomposition method. The experimental results show that our
method improves the sample efficiency in different partially observable Markov
decision process domains.

本文提出了一种隐式的基于模型的多智能体强化学习方法，基于价值分解方法，能够提高在不同部分可观察的马尔可夫决策过程领域中的样本效率。