Cooperative game is a critical research area in multi-agent reinforcement learning (MARL). Global reward game is a subclass of cooperative games, where all agents aim to maximize cumulative global rewards. Credit assignment is an important problem studied in the global reward game. Most works stand by the view of non-cooperative-game theoretical framework with the shared reward approach, i.e., each agent is assigned a shared global reward directly. This, however, may give each agent an inaccurate feedback on his contribution to the group. In this paper, we introduce a cooperative-game theoretical framework and extend it to the finite-horizon case. We show that our proposed framework is a superset of the global reward game. Based on this framework, we propose an algorithm called Shapley Q-value policy gradient (SQPG) to learn a local reward approach that can distribute the cumulative global reward fairly, reflecting each agent's own contribution in contrast to the shared reward approach. We evaluate our method on the Cooperative Navigation, Prey-and-Predator and Traffic Junction, compared with MADDPG, COMA, Independent actor-critic and Independent DDPG. In the experiments, our algorithm shows better convergence than the baselines.

本研究介绍了一个合作游戏理论框架(扩展凸性博弈ECG)，提出了一个局部奖励方法(Shapley Q-value)，并用其作为每个代理的评估函数，提出了一个MARL算法SQDDPG，并在导航，猎物与捕食者和交通路口等环境进行了实验，结果显示其相较于目前现有算法具有更快的收敛速度和公平的分配。

Shapley Q-value：一种解决全局奖励博弈的本地奖励方法