Establishing causal relationships between actions and outcomes is fundamental
for accountable multi-agent decision-making. However, interpreting and
quantifying agents' contributions to such relationships pose significant
challenges. These challenges are particularly prominent in the context of
multi-agent sequential decision-making, where the causal effect of an agent's
action on the outcome depends on how the other agents respond to that action.
In this paper, our objective is to present a systematic approach for
attributing the causal effects of agents' actions to the influence they exert
on other agents. Focusing on multi-agent Markov decision processes, we
introduce agent-specific effects (ASE), a novel causal quantity that measures
the effect of an agent's action on the outcome that propagates through other
agents. We then turn to the counterfactual counterpart of ASE (cf-ASE), provide
a sufficient set of conditions for identifying cf-ASE, and propose a practical
sampling-based algorithm for estimating it. Finally, we experimentally evaluate
the utility of cf-ASE through a simulation-based testbed, which includes a
sepsis management environment.

本文介绍了一种系统的方法来将代理的行为的因果效果归因于其对其他代理的影响，通过引入代理特定效应（ASE）作为一种衡量代理行为对其他代理传播的结果影响的新型因果量。同时，我们提出了一种基于抽样的算法来估计 ASE 的反事实对应物（cf-ASE），并通过模拟测试来实验评估 cf-ASE 的实用性，其中包括一种败血症管理环境。

特定代理作用

Agent-Specific Effects

The paper considers independent reinforcement learning (IRL) for multi-agent
decision-making process in the paradigm of federated learning (FL). We show
that FL can clearly improve the policy performance of IRL in terms of training
efficiency and stability. However, since the policy parameters are trained
locally and aggregated iteratively through a central server in FL, frequent
information exchange incurs a large amount of communication overheads. To reach
a good balance between improving the model's convergence performance and
reducing the required communication and computation overheads, this paper
proposes a system utility function and develops a consensus-based optimization
scheme on top of the periodic averaging method, which introduces the consensus
algorithm into FL for the exchange of a model's local gradients. This paper
also provides novel convergence guarantees for the developed method, and
demonstrates its superior effectiveness and efficiency in improving the system
utility value through theoretical analyses and numerical simulation results.

本文考虑了联邦学习范式下的独立强化学习问题，提出了一种系统效用函数以及基于周期平均法的共识优化方案，在理论分析与数值模拟中证明了其优异性和有效性。

面向联邦强化学习的通信高效共识机制

Communication-Efficient Consensus Mechanism for Federated Reinforcement Learning

We consider finite and infinite horizon dynamic programming problems, where
the control at each stage consists of several distinct decisions, each one made
by one of several agents. We introduce an approach, whereby at every stage,
each agent's decision is made by executing a local rollout algorithm that uses
a base policy, together with some coordinating information from the other
agents. The amount of local computation required at every stage by each agent
is independent of the number of agents, while the amount of total computation
(over all agents) grows linearly with the number of agents. By contrast, with
the standard rollout algorithm, the amount of total computation grows
exponentially with the number of agents. Despite the drastic reduction in
required computation, we show that our algorithm has the fundamental cost
improvement property of rollout: an improved performance relative to the base
policy. We also discuss possibilities to improve further the method's
computational efficiency through limited agent coordination and parallelization
of the agents' computations. Finally, we explore related approximate policy
iteration algorithms for infinite horizon problems, and we prove that the cost
improvement property steers the algorithm towards convergence to an
agent-by-agent optimal policy.

本文介绍了一种基于本地回滚算法的多智能体决策方法，该方法通过配合其他智能体的协调信息，在每个阶段使每个智能体的决策都能够独立计算，并且与智能体数量无关。此外，我们探讨了并行化智能体计算和近似策略迭代算法等方法来进一步提高本方法的计算效率，并证明了本文算法可以收敛到智能体的最优策略。