Measuring the contribution of individual agents is challenging in cooperative multi-agent reinforcement learning (MARL). In cooperative MARL, team performance is typically inferred from a single shared global reward. Arguably, among the best current approaches to effectively measure individual agent contributions is to use Shapley values. However, calculating these values is expensive as the computational complexity grows exponentially with respect to the number of agents. In this paper, we adapt difference rewards into an efficient method for quantifying the contribution of individual agents, referred to as Agent Importance, offering a linear computational complexity relative to the number of agents. We show empirically that the computed values are strongly correlated with the true Shapley values, as well as the true underlying individual agent rewards, used as the ground truth in environments where these are available. We demonstrate how Agent Importance can be used to help study MARL systems by diagnosing algorithmic failures discovered in prior MARL benchmarking work. Our analysis illustrates Agent Importance as a valuable explainability component for future MARL benchmarks.

合作多智能体强化学习中个体代理的贡献度一直是一项具有挑战性的任务，而本文提出的Agent Importance方法通过计算个体代理的贡献度，其计算复杂度相对于代理数量呈线性增长，能够有效代替耗时的Shapley values方法。实证结果表明Agent Importance与真实的Shapley values及底层个体代理奖励具有较强相关性，并且可用于诊断多智能体强化学习系统中的算法失败情况，从而作为未来强化学习基准测试的一个有价值的可解释性组成部分。

高效量化合作 MARL 中个体代理的重要性