Learning a world model for model-free Reinforcement Learning (RL) agents can
significantly improve the sample efficiency by learning policies in
imagination. However, building a world model for Multi-Agent RL (MARL) can be
particularly challenging due to the scalability issue in a centralized
architecture arising from a large number of agents, and also the
non-stationarity issue in a decentralized architecture stemming from the
inter-dependency among agents. To address both challenges, we propose a novel
world model for MARL that learns decentralized local dynamics for scalability,
combined with a centralized representation aggregation from all agents. We cast
the dynamics learning as an auto-regressive sequence modeling problem over
discrete tokens by leveraging the expressive Transformer architecture, in order
to model complex local dynamics across different agents and provide accurate
and consistent long-term imaginations. As the first pioneering
Transformer-based world model for multi-agent systems, we introduce a Perceiver
Transformer as an effective solution to enable centralized representation
aggregation within this context. Results on Starcraft Multi-Agent Challenge
(SMAC) show that it outperforms strong model-free approaches and existing
model-based methods in both sample efficiency and overall performance.

为了解决多智能体强化学习中的可扩展性和非稳态性问题，我们提出了一种新的多智能体世界模型，通过学习分散的局部动力学来实现可扩展性，并通过中心化表示的聚合来解决非稳态性问题。我们引入了 Perceiver Transformer 作为一种有效的解决方案，通过 Transformer 体系结构对复杂的局部动力学进行建模，从而提供准确而一致的长期想象。在 Starcraft Multi-Agent Challenge（SMAC）上的结果显示，该模型在样本效率和整体性能方面优于强化学习和现有的基于模型的方法。