The necessity for cooperation among intelligent machines has popularised
cooperative multi-agent reinforcement learning (MARL) in the artificial
intelligence (AI) research community. However, many research endeavors have
been focused on developing practical MARL algorithms whose effectiveness has
been studied only empirically, thereby lacking theoretical guarantees. As
recent studies have revealed, MARL methods often achieve performance that is
unstable in terms of reward monotonicity or suboptimal at convergence. To
resolve these issues, in this paper, we introduce a novel framework named
Heterogeneous-Agent Mirror Learning (HAML) that provides a general template for
MARL algorithmic designs. We prove that algorithms derived from the HAML
template satisfy the desired properties of the monotonic improvement of the
joint reward and the convergence to Nash equilibrium. We verify the
practicality of HAML by proving that the current state-of-the-art cooperative
MARL algorithms, HATRPO and HAPPO, are in fact HAML instances. Next, as a
natural outcome of our theory, we propose HAML extensions of two well-known RL
algorithms, HAA2C (for A2C) and HADDPG (for DDPG), and demonstrate their
effectiveness against strong baselines on StarCraftII and Multi-Agent MuJoCo
tasks.

本文介绍了一种名为异构代理镜像学习（HAML）的新型框架，该框架提供了一种通用的 MARL 算法设计模板，解决了在奖励单调性或收敛时的非最优性能问题，通过证明来自 HAML 模板的算法满足单调改进联合奖励和收敛到纳什均衡的期望属性，并通过在星际争霸 II 和多智能体 MuJoCo 任务中验证了 HAML 的实用性。