Mean Field Control (MFC) is a powerful approximation tool to solve
large-scale Multi-Agent Reinforcement Learning (MARL) problems. However, the
success of MFC relies on the presumption that given the local states and
actions of all the agents, the next (local) states of the agents evolve
conditionally independent of each other. Here we demonstrate that even in a
MARL setting where agents share a common global state in addition to their
local states evolving conditionally independently (thus introducing a
correlation between the state transition processes of individual agents), the
MFC can still be applied as a good approximation tool. The global state is
assumed to be non-decomposable i.e., it cannot be expressed as a collection of
local states of the agents. We compute the approximation error as
$\mathcal{O}(e)$ where $e=\frac{1}{\sqrt{N}}\left[\sqrt{|\mathcal{X}|}
+\sqrt{|\mathcal{U}|}\right]$. The size of the agent population is denoted by
the term $N$, and $|\mathcal{X}|, |\mathcal{U}|$ respectively indicate the
sizes of (local) state and action spaces of individual agents. The
approximation error is found to be independent of the size of the shared global
state space. We further demonstrate that in a special case if the reward and
state transition functions are independent of the action distribution of the
population, then the error can be improved to
$e=\frac{\sqrt{|\mathcal{X}|}}{\sqrt{N}}$. Finally, we devise a Natural Policy
Gradient based algorithm that solves the MFC problem with
$\mathcal{O}(\epsilon^{-3})$ sample complexity and obtains a policy that is
within $\mathcal{O}(\max\{e,\epsilon\})$ error of the optimal MARL policy for
any $\epsilon>0$.

本文研究了在多智能体强化学习中，基于平均场控制的近似算法可以在存在全局状态的情况下进行有效求解，并提出了基于自然策略梯度算法可以在小样本情况下求解此问题，并得到了接近最优解的策略.

存在不可分解的共享全局状态时的基于均场控制的多智能体强化学习近似

Mean-Field Control based Approximation of Multi-Agent Reinforcement Learning in Presence of a Non-decomposable Shared Global State

BERT is a popular language model whose main pre-training task is to fill in
the blank, i.e., predicting a word that was masked out of a sentence, based on
the remaining words. In some applications, however, having an additional
context can help the model make the right prediction, e.g., by taking the
domain or the time of writing into account. This motivates us to advance the
BERT architecture by adding a global state for conditioning on a fixed-sized
context. We present our two novel approaches and apply them to an industry
use-case, where we complete fashion outfits with missing articles, conditioned
on a specific customer. An experimental comparison to other methods from the
literature shows that our methods improve personalization significantly.

该论文提出了两种新的全局状态方法，将其应用到时尚搭配领域，通过具体客户特征进行服装搭配，实验比较表明这种方法能显著提高个性化推荐的效果。