Markov Potential Games (MPGs) form an important sub-class of Markov games, which are a common framework to model multi-agent reinforcement learning problems. In particular, MPGs include as a special case the identical-interest setting where all the agents share the same reward function. Scaling the performance of Nash equilibrium learning algorithms to a large number of agents is crucial for multi-agent systems. To address this important challenge, we focus on the independent learning setting where agents can only have access to their local information to update their own policy. In prior work on MPGs, the iteration complexity for obtaining $\epsilon$-Nash regret scales linearly with the number of agents $N$. In this work, we investigate the iteration complexity of an independent policy mirror descent (PMD) algorithm for MPGs. We show that PMD with KL regularization, also known as natural policy gradient, enjoys a better $\sqrt{N}$ dependence on the number of agents, improving over PMD with Euclidean regularization and prior work. Furthermore, the iteration complexity is also independent of the sizes of the agents' action spaces.

本文解决了多智能体强化学习中马尔可夫潜在博弈（MPGs）的大规模学习问题，尤其是在相同利益设置下的纳什均衡学习。提出的独立策略镜像下降算法(PMD)通过KL正则化，实现了迭代复杂度对智能体数量的依赖降低到$\sqrt{N}$，显著优于以往研究，实现了在大规模智能体系统中的高效学习。

马尔可夫潜在博弈的独立策略镜像下降：扩展到大数量玩家