Effective agent coordination is crucial in cooperative Multi-Agent
Reinforcement Learning (MARL). While agent cooperation can be represented by
graph structures, prevailing graph learning methods in MARL are limited. They
rely solely on one-step observations, neglecting crucial historical
experiences, leading to deficient graphs that foster redundant or detrimental
information exchanges. Additionally, high computational demands for action-pair
calculations in dense graphs impede scalability. To address these challenges,
we propose inferring a Latent Temporal Sparse Coordination Graph (LTS-CG) for
MARL. The LTS-CG leverages agents' historical observations to calculate an
agent-pair probability matrix, where a sparse graph is sampled from and used
for knowledge exchange between agents, thereby simultaneously capturing agent
dependencies and relation uncertainty. The computational complexity of this
procedure is only related to the number of agents. This graph learning process
is further augmented by two innovative characteristics: Predict-Future, which
enables agents to foresee upcoming observations, and Infer-Present, ensuring a
thorough grasp of the environmental context from limited data. These features
allow LTS-CG to construct temporal graphs from historical and real-time
information, promoting knowledge exchange during policy learning and effective
collaboration. Graph learning and agent training occur simultaneously in an
end-to-end manner. Our demonstrated results on the StarCraft II benchmark
underscore LTS-CG's superior performance.

在合作多智能体强化学习中，有效的智能体协调至关重要。为了解决现有方法中对历史经验的忽视和稠密图计算的可扩展性问题，我们提出了一种基于潜在时间稀疏协调图的多智能体强化学习方法。该方法利用智能体的历史观测计算智能体对概率矩阵，并基于此矩阵生成稀疏图，以促进智能体之间的知识交流，同时捕捉智能体之间的依赖关系和关系不确定性。该方法还引入了 “预测未来” 和 “推断现在” 两个创新特性，使得该方法能够从有限数据中构建历史和实时信息的时间图，促进策略学习和有效协作。实验结果表明，该方法在 StarCraft II 基准测试中具有卓越的性能。