Recent advances in multi-agent reinforcement learning have been largely
limited in training one model from scratch for every new task. The limitation
is due to the restricted model architecture related to fixed input and output
dimensions. This hinders the experience accumulation and transfer of the
learned agent over tasks with diverse levels of difficulty (e.g. 3 vs 3 or 5 vs
6 multi-agent games). In this paper, we make the first attempt to explore a
universal multi-agent reinforcement learning pipeline, designing one single
architecture to fit tasks with the requirement of different observation and
action configurations. Unlike previous RNN-based models, we utilize a
transformer-based model to generate a flexible policy by decoupling the policy
distribution from the intertwined input observation with an importance weight
measured by the merits of the self-attention mechanism. Compared to a standard
transformer block, the proposed model, named as Universal Policy Decoupling
Transformer (UPDeT), further relaxes the action restriction and makes the
multi-agent task's decision process more explainable. UPDeT is general enough
to be plugged into any multi-agent reinforcement learning pipeline and equip
them with strong generalization abilities that enables the handling of multiple
tasks at a time. Extensive experiments on large-scale SMAC multi-agent
competitive games demonstrate that the proposed UPDeT-based multi-agent
reinforcement learning achieves significant results relative to
state-of-the-art approaches, demonstrating advantageous transfer capability in
terms of both performance and training speed (10 times faster).

本文提出了一种称为 Universal Policy Decoupling Transformer (UPDeT) 的 transformer-based 模型，可插入任何多智能体强化学习管道并具备强大的泛化能力，证明了相对于现有方法具有优异的迁移能力，在大规模 SMAC 多智能体竞争游戏中取得了显著的表现和训练速度（快 10 倍）。