Measuring the contribution of individual agents is challenging in cooperative
multi-agent reinforcement learning (MARL). In cooperative MARL, team
performance is typically inferred from a single shared global reward. Arguably,
among the best current approaches to effectively measure individual agent
contributions is to use Shapley values. However, calculating these values is
expensive as the computational complexity grows exponentially with respect to
the number of agents. In this paper, we adapt difference rewards into an
efficient method for quantifying the contribution of individual agents,
referred to as Agent Importance, offering a linear computational complexity
relative to the number of agents. We show empirically that the computed values
are strongly correlated with the true Shapley values, as well as the true
underlying individual agent rewards, used as the ground truth in environments
where these are available. We demonstrate how Agent Importance can be used to
help study MARL systems by diagnosing algorithmic failures discovered in prior
MARL benchmarking work. Our analysis illustrates Agent Importance as a valuable
explainability component for future MARL benchmarks.

合作多智能体强化学习中个体代理的贡献度一直是一项具有挑战性的任务，而本文提出的 Agent Importance 方法通过计算个体代理的贡献度，其计算复杂度相对于代理数量呈线性增长，能够有效代替耗时的 Shapley values 方法。实证结果表明 Agent Importance 与真实的 Shapley values 及底层个体代理奖励具有较强相关性，并且可用于诊断多智能体强化学习系统中的算法失败情况，从而作为未来强化学习基准测试的一个有价值的可解释性组成部分。

高效量化合作 MARL 中个体代理的重要性

Efficiently Quantifying Individual Agent Importance in Cooperative MARL

Multi-agent reinforcement learning (MARL) suffers from the non-stationarity
problem, which is the ever-changing targets at every iteration when multiple
agents update their policies at the same time. Starting from first principle,
in this paper, we manage to solve the non-stationarity problem by proposing
bidirectional action-dependent Q-learning (ACE). Central to the development of
ACE is the sequential decision-making process wherein only one agent is allowed
to take action at one time. Within this process, each agent maximizes its value
function given the actions taken by the preceding agents at the inference
stage. In the learning phase, each agent minimizes the TD error that is
dependent on how the subsequent agents have reacted to their chosen action.
Given the design of bidirectional dependency, ACE effectively turns a
multiagent MDP into a single-agent MDP. We implement the ACE framework by
identifying the proper network representation to formulate the action
dependency, so that the sequential decision process is computed implicitly in
one forward pass. To validate ACE, we compare it with strong baselines on two
MARL benchmarks. Empirical experiments demonstrate that ACE outperforms the
state-of-the-art algorithms on Google Research Football and StarCraft
Multi-Agent Challenge by a large margin. In particular, on SMAC tasks, ACE
achieves 100% success rate on almost all the hard and super-hard maps. We
further study extensive research problems regarding ACE, including extension,
generalization, and practicability. Code is made available to facilitate
further research.

此研究提出了双向依赖 Q-learning（ACE）方法，通过设计正确的网络表示，在顺序决策过程中隐式计算以解决多智能体强化学习中的非静态问题，并通过比较实践验证 ACE 超越了 Google Research Football 和 StarCraft 多智能体挑战中其他算法。