The exploitation of extra state information has been an active research area
in multi-agent reinforcement learning (MARL). QMIX represents the joint
action-value using a non-negative function approximator and achieves the best
performance, by far, on multi-agent benchmarks, StarCraft II micromanagement
tasks. However, our experiments show that, in some cases, QMIX is incompatible
with A2C, a training paradigm that promotes algorithm training efficiency. To
obtain a reasonable trade-off between training efficiency and algorithm
performance, we extend value-decomposition to actor-critics that are compatible
with A2C and propose a novel actor-critic framework, value-decomposition
actor-critics (VDACs). We evaluate VDACs on the testbed of StarCraft II
micromanagement tasks and demonstrate that the proposed framework improves
median performance over other actor-critic methods. Furthermore, we use a set
of ablation experiments to identify the key factors that contribute to the
performance of VDACs.

在多智能体强化学习中，QMIX 使用非负函数逼近器表示联合动作价值，并在 StarCraft II 微观管理任务中取得了迄今最好的性能。然而，我们的实验表明，在某些情况下，QMIX 与 A2C 不兼容。为了在培训效率和算法性能之间获得合理的折衷，我们将价值分解扩展到与 A2C 兼容的演员临界值，并提出了一种新的演员临界值框架，即价值分解演员临界值（VDACs）。我们在 StarCraft II 微观管理任务测试平台上评估 VDACs，并证明所提出的框架改进了其他演员临界值方法的中位性能。此外，我们使用一组消融实验来确定对 VDACs 的性能做出贡献的关键因素。