Multi-Agent Reinforcement Learning (MARL) has been successful in solving many
cooperative challenges. However, classic non-hierarchical MARL algorithms still
cannot address various complex multi-agent problems that require hierarchical
cooperative behaviors. The cooperative knowledge and policies learned in
non-hierarchical algorithms are implicit and not interpretable, thereby
restricting the integration of existing knowledge. This paper proposes a novel
hierarchical MARL model called Hierarchical Cooperation Graph Learning (HCGL)
for solving general multi-agent problems. HCGL has three components: a dynamic
Extensible Cooperation Graph (ECG) for achieving self-clustering cooperation; a
group of graph operators for adjusting the topology of ECG; and an MARL
optimizer for training these graph operators. HCGL's key distinction from other
MARL models is that the behaviors of agents are guided by the topology of ECG
instead of policy neural networks. ECG is a three-layer graph consisting of an
agent node layer, a cluster node layer, and a target node layer. To manipulate
the ECG topology in response to changing environmental conditions, four graph
operators are trained to adjust the edge connections of ECG dynamically. The
hierarchical feature of ECG provides a unique approach to merge primitive
actions (actions executed by the agents) and cooperative actions (actions
executed by the clusters) into a unified action space, allowing us to integrate
fundamental cooperative knowledge into an extensible interface. In our
experiments, the HCGL model has shown outstanding performance in multi-agent
benchmarks with sparse rewards. We also verify that HCGL can easily be
transferred to large-scale scenarios with high zero-shot transfer success
rates.

提出了一种名为 Hierarchical Cooperation Graph Learning (HCGL) 的新型分层 MARL 模型，可以解决复杂的多智能体问题，通过动态可扩展合作图 (ECG) 实现自我聚类合作，调整 ECG 拓扑结构的图操作符，以及训练这些图操作符的 MARL 优化器。实验结果表明，HCGL 模型在稀疏奖励的多智能体基准测试中表现出色，并可以轻松应用于具有高零 - shot 传输成功率的大规模场景。