The field of multi-agent reinforcement learning (MARL) has made considerable
progress towards controlling challenging multi-agent systems by employing
various learning methods. Numerous of these approaches focus on empirical and
algorithmic aspects of the MARL problems and lack a rigorous theoretical
foundation. Graphon mean field games (GMFGs) on the other hand provide a
scalable and mathematically well-founded approach to learning problems that
involve a large number of connected agents. In standard GMFGs, the connections
between agents are undirected, unweighted and invariant over time. Our paper
introduces colored digraphon mean field games (CDMFGs) which allow for weighted
and directed links between agents that are also adaptive over time. Thus,
CDMFGs are able to model more complex connections than standard GMFGs. Besides
a rigorous theoretical analysis including both existence and convergence
guarantees, we provide a learning scheme and illustrate our findings with an
epidemics model and a model of the systemic risk in financial markets.

介绍了一种基于有向带权图的多智体强化学习模型，可模拟金融市场中的系统性风险。

通过带权重有向图的彩色有向图来研究均场博弈

Mean Field Games on Weighted and Directed Graphs via Colored Digraphons

Although the field of multi-agent reinforcement learning (MARL) has made
considerable progress in the last years, solving systems with a large number of
agents remains a hard challenge. Graphon mean field games (GMFGs) enable the
scalable analysis of MARL problems that are otherwise intractable. By the
mathematical structure of graphons, this approach is limited to dense graphs
which are insufficient to describe many real-world networks such as power law
graphs. Our paper introduces a novel formulation of GMFGs, called LPGMFGs,
which leverages the graph theoretical concept of $L^p$ graphons and provides a
machine learning tool to efficiently and accurately approximate solutions for
sparse network problems. This especially includes power law networks which are
empirically observed in various application areas and cannot be captured by
standard graphons. We derive theoretical existence and convergence guarantees
and give empirical examples that demonstrate the accuracy of our learning
approach for systems with many agents. Furthermore, we extend the Online Mirror
Descent (OMD) learning algorithm to our setup to accelerate learning speed,
empirically show its capabilities, and conduct a theoretical analysis using the
novel concept of smoothed step graphons. In general, we provide a scalable,
mathematically well-founded machine learning approach to a large class of
otherwise intractable problems of great relevance in numerous research fields.

本文提出了一种名为 LPGMFGs 的图论概念 $L^p$ graphons 的解法，该方法可以有效而准确地解决大型多智能体强化学习问题，特别是那些包括幂律网络等能用标准图论难以描述的问题。

学习稀疏图形均场博弈

Learning Sparse Graphon Mean Field Games

Recent advances at the intersection of dense large graph limits and mean
field games have begun to enable the scalable analysis of a broad class of
dynamical sequential games with large numbers of agents. So far, results have
been largely limited to graphon mean field systems with continuous-time
diffusive or jump dynamics, typically without control and with little focus on
computational methods. We propose a novel discrete-time formulation for graphon
mean field games as the limit of non-linear dense graph Markov games with weak
interaction. On the theoretical side, we give extensive and rigorous existence
and approximation properties of the graphon mean field solution in sufficiently
large systems. On the practical side, we provide general learning schemes for
graphon mean field equilibria by either introducing agent equivalence classes
or reformulating the graphon mean field system as a classical mean field
system. By repeatedly finding a regularized optimal control solution and its
generated mean field, we successfully obtain plausible approximate Nash
equilibria in otherwise infeasible large dense graph games with many agents.
Empirically, we are able to demonstrate on a number of examples that the
finite-agent behavior comes increasingly close to the mean field behavior for
our computed equilibria as the graph or system size grows, verifying our
theory. More generally, we successfully apply policy gradient reinforcement
learning in conjunction with sequential Monte Carlo methods.

以非线性稠密图马尔可夫游戏为极限，提出了图分块场博弈的新离散时间公式，并通过正则化最优控制解和其生成的平均场重复发现策略梯度加强学习，成功获得在众多玩家的场景中可行的近似纳什均衡。