The necessity for cooperation among intelligent machines has popularised
cooperative multi-agent reinforcement learning (MARL) in AI research. However,
many research endeavours heavily rely on parameter sharing among agents, which
confines them to only homogeneous-agent setting and leads to training
instability and lack of convergence guarantees. To achieve effective
cooperation in the general heterogeneous-agent setting, we propose
Heterogeneous-Agent Reinforcement Learning (HARL) algorithms that resolve the
aforementioned issues. Central to our findings are the multi-agent advantage
decomposition lemma and the sequential update scheme. Based on these, we
develop the provably correct Heterogeneous-Agent Trust Region Learning (HATRL)
that is free of parameter-sharing constraint, and derive HATRPO and HAPPO by
tractable approximations. Furthermore, we discover a novel framework named
Heterogeneous-Agent Mirror Learning (HAML), which strengthens theoretical
guarantees for HATRPO and HAPPO and provides a general template for cooperative
MARL algorithmic designs. We prove that all algorithms derived from HAML
inherently enjoy monotonic improvement of joint reward and convergence to Nash
Equilibrium. As its natural outcome, HAML validates more novel algorithms in
addition to HATRPO and HAPPO, including HAA2C, HADDPG, and HATD3, which
consistently outperform their existing MA-counterparts. We comprehensively test
HARL algorithms on six challenging benchmarks and demonstrate their superior
effectiveness and stability for coordinating heterogeneous agents compared to
strong baselines such as MAPPO and QMIX.

本文提出了基于 HARL 算法的新框架 HAML，将多智能体强化学习的合作扩展到异构智能体模式，并对该框架下的多种算法进行了验证和比较。测试表明，HARL 算法在协调异构智能体方面的稳定性和有效性要优于现有的 MA 对应物。