This paper considers multi-agent reinforcement learning (MARL) in networked
system control. Specifically, each agent learns a decentralized control policy
based on local observations and messages from connected neighbors. We formulate
such a networked MARL (NMARL) problem as a spatiotemporal Markov decision
process and introduce a spatial discount factor to stabilize the training of
each local agent. Further, we propose a new differentiable communication
protocol, called NeurComm, to reduce information loss and non-stationarity in
NMARL. Based on experiments in realistic NMARL scenarios of adaptive traffic
signal control and cooperative adaptive cruise control, an appropriate spatial
discount factor effectively enhances the learning curves of non-communicative
MARL algorithms, while NeurComm outperforms existing communication protocols in
both learning efficiency and control performance.

本文研究了网络系统控制中的多智能体强化学习问题，提出了基于空间折扣因子的 NMARL 问题并引入了一种可微的通信协议 NeurComm 以提高学习效率和控制性能。实验结果表明，合适的空间折扣因子可以有效提高 MARL 算法的非通讯性学习曲线，而 NeurComm 在学习效率和控制性能方面均优于现有的通信协议。