Multi-Agent Policy Gradient (MAPG) has made significant progress in recent
years. However, centralized critics in state-of-the-art MAPG methods still face
the centralized-decentralized mismatch (CDM) issue, which means sub-optimal
actions by some agents will affect other agent's policy learning. While using
individual critics for policy updates can avoid this issue, they severely limit
cooperation among agents. To address this issue, we propose an agent topology
framework, which decides whether other agents should be considered in policy
gradient and achieves compromise between facilitating cooperation and
alleviating the CDM issue. The agent topology allows agents to use coalition
utility as learning objective instead of global utility by centralized critics
or local utility by individual critics. To constitute the agent topology,
various models are studied. We propose Topology-based multi-Agent Policy
gradiEnt (TAPE) for both stochastic and deterministic MAPG methods. We prove
the policy improvement theorem for stochastic TAPE and give a theoretical
explanation for the improved cooperation among agents. Experiment results on
several benchmarks show the agent topology is able to facilitate agent
cooperation and alleviate CDM issue respectively to improve performance of
TAPE. Finally, multiple ablation studies and a heuristic graph search algorithm
are devised to show the efficacy of the agent topology.

提出了一个代理拓扑框架，通过在策略梯度中考虑其他代理来实现协作与解决分布一致性不匹配问题的折中方案。该代理拓扑可以使代理使用联盟效用作为学习目标，避免了全局效用或局部效用带来的问题，并通过实验结果表明能够改善 TAPE 的性能。