We propose a novel framework for value function factorization in multi-agent
deep reinforcement learning (MARL) using graph neural networks (GNNs). In
particular, we consider the team of agents as the set of nodes of a complete
directed graph, whose edge weights are governed by an attention mechanism.
Building upon this underlying graph, we introduce a mixing GNN module, which is
responsible for i) factorizing the team state-action value function into
individual per-agent observation-action value functions, and ii) explicit
credit assignment to each agent in terms of fractions of the global team
reward. Our approach, which we call GraphMIX, follows the centralized training
and decentralized execution paradigm, enabling the agents to make their
decisions independently once training is completed. We show the superiority of
GraphMIX as compared to the state-of-the-art on several scenarios in the
StarCraft II multi-agent challenge (SMAC) benchmark. We further demonstrate how
GraphMIX can be used in conjunction with a recent hierarchical MARL
architecture to both improve the agents' performance and enable fine-tuning
them on mismatched test scenarios with higher numbers of agents and/or actions.

该论文提出了一种新颖的基于图神经网络的多智能体深度强化学习价值函数分解框架，包括目标团队代理作为一组完整有向图的节点、采用注意机制进行边权重控制、推出混合 GNN 模块用于把团队状态 - 动作价值函数分解为单独智能体的观测 - 动作价值函数、显式接受损失分配。该方法称为 GraphMIX，能够优于当前最先进方法，可用于 StarCraft II 多智能体挑战基准测试中，同时能够改善智能体性能并使其适应更高数量和 / 或操作的不匹配测试情景。