In many real-world tasks, multiple agents must learn to coordinate with each
other given their private observations and limited communication ability. Deep
multiagent reinforcement learning (Deep-MARL) algorithms have shown superior
performance in such challenging settings. One representative class of work is
multiagent value decomposition, which decomposes the global shared multiagent
Q-value $Q_{tot}$ into individual Q-values $Q^{i}$ to guide individuals'
behaviors, i.e. VDN imposing an additive formation and QMIX adopting a
monotonic assumption using an implicit mixing method. However, most of the
previous efforts impose certain assumptions between $Q_{tot}$ and $Q^{i}$ and
lack theoretical groundings. Besides, they do not explicitly consider the
agent-level impact of individuals to the whole system when transforming
individual $Q^{i}$s into $Q_{tot}$. In this paper, we theoretically derive a
general formula of $Q_{tot}$ in terms of $Q^{i}$, based on which we can
naturally implement a multi-head attention formation to approximate $Q_{tot}$,
resulting in not only a refined representation of $Q_{tot}$ with an agent-level
attention mechanism, but also a tractable maximization algorithm of
decentralized policies. Extensive experiments demonstrate that our method
outperforms state-of-the-art MARL methods on the widely adopted StarCraft
benchmark across different scenarios, and attention analysis is further
conducted with valuable insights.

本文提出了一种基于多头注意力机制和 Q-value 分解的深度多智能体强化学习算法和最大化算法，并在 StarCraft 基准测试中表现出国内领先的性能。