Role-based learning is a promising approach to improving the performance of
Multi-Agent Reinforcement Learning (MARL). Nevertheless, without manual
assistance, current role-based methods cannot guarantee stably discovering a
set of roles to effectively decompose a complex task, as they assume either a
predefined role structure or practical experience for selecting
hyperparameters. In this article, we propose a mathematical Structural
Information principles-based Role Discovery method, namely SIRD, and then
present a SIRD optimizing MARL framework, namely SR-MARL, for multi-agent
collaboration. The SIRD transforms role discovery into a hierarchical action
space clustering. Specifically, the SIRD consists of structuralization,
sparsification, and optimization modules, where an optimal encoding tree is
generated to perform abstracting to discover roles. The SIRD is agnostic to
specific MARL algorithms and flexibly integrated with various value function
factorization approaches. Empirical evaluations on the StarCraft II
micromanagement benchmark demonstrate that, compared with state-of-the-art MARL
algorithms, the SR-MARL framework improves the average test win rate by 0.17%,
6.08%, and 3.24%, and reduces the deviation by 16.67%, 30.80%, and 66.30%,
under easy, hard, and super hard scenarios.

本文提出了一种数学结构化信息原则为基础的角色发现方法 ——SIRD，以及一个基于 SIRD 优化的多智能体协作框架 ——SR-MARL，并在 StarCraft II 微观管理基准测试中证明，SR-MARL 框架相对于最先进的 MARL 算法在易、难和超级难的情况下平均测试胜率分别提高了 0.17%、6.08% 和 3.24%，减少了 16.67%、30.80% 和 66.30% 的偏差。

结构信息原则下的有效和稳定的基于角色的多智能体协作

Effective and Stable Role-Based Multi-Agent Collaboration by Structural  Information Principles

Role-based learning holds the promise of achieving scalable multi-agent
learning by decomposing complex tasks using roles. However, it is largely
unclear how to efficiently discover such a set of roles. To solve this problem,
we propose to first decompose joint action spaces into restricted role action
spaces by clustering actions according to their effects on the environment and
other agents. Learning a role selector based on action effects makes role
discovery much easier because it forms a bi-level learning hierarchy -- the
role selector searches in a smaller role space and at a lower temporal
resolution, while role policies learn in significantly reduced primitive
action-observation spaces. We further integrate information about action
effects into the role policies to boost learning efficiency and policy
generalization. By virtue of these advances, our method (1) outperforms the
current state-of-the-art MARL algorithms on 10 of the 14 scenarios that
comprise the challenging StarCraft II micromanagement benchmark and (2)
achieves rapid transfer to new environments with three times the number of
agents. Demonstrative videos are available at
this https URL .

通过聚类行动效果、学习角色选择器和角色策略，使角色发现更加容易并提高学习效率和策略泛化能力，该方法在 StarCraft II 多智能体微观管理基准测试的 14 个场景中表现优秀，实现了快速迁移至三倍数量的代理。