Recently, deep multiagent reinforcement learning (MARL) has become a highly
active research area as many real-world problems can be inherently viewed as
multiagent systems. A particularly interesting and widely applicable class of
problems is the partially observable cooperative multiagent setting, in which a
team of agents learns to coordinate their behaviors conditioning on their
private observations and commonly shared global reward signals. One natural
solution is to resort to the centralized training and decentralized execution
paradigm. During centralized training, one key challenge is the multiagent
credit assignment: how to allocate the global rewards for individual agent
policies for better coordination towards maximizing system-level's benefits. In
this paper, we propose a new method called Q-value Path Decomposition (QPD) to
decompose the system's global Q-values into individual agents' Q-values. Unlike
previous works which restrict the representation relation of the individual
Q-values and the global one, we leverage the integrated gradient attribution
technique into deep MARL to directly decompose global Q-values along trajectory
paths to assign credits for agents. We evaluate QPD on the challenging
StarCraft II micromanagement tasks and show that QPD achieves the
state-of-the-art performance in both homogeneous and heterogeneous multiagent
scenarios compared with existing cooperative MARL algorithms.

本论文提出一种名为 Q-value Path Decomposition（QPD）的方法，该方法利用集成渐变归因技术对深度多智能体强化学习中的全局 Q 值进行分解，以便为代理分配信用，解决了多智能体信用分配的关键挑战，该方法在 StarCraft II 微观管理任务中优于现有协作 MARL 算法。