Centralized training with decentralized execution (CTDE) is a widely-used
learning paradigm that has achieved significant success in complex tasks.
However, partial observability issues and the absence of effectively shared
signals between agents often limit its effectiveness in fostering cooperation.
While communication can address this challenge, it simultaneously reduces the
algorithm's practicality. Drawing inspiration from human team cooperative
learning, we propose a novel paradigm that facilitates a gradual shift from
explicit communication to tacit cooperation. In the initial training stage, we
promote cooperation by sharing relevant information among agents and
concurrently reconstructing this information using each agent's local
trajectory. We then combine the explicitly communicated information with the
reconstructed information to obtain mixed information. Throughout the training
process, we progressively reduce the proportion of explicitly communicated
information, facilitating a seamless transition to fully decentralized
execution without communication. Experimental results in various scenarios
demonstrate that the performance of our method without communication can
approaches or even surpasses that of QMIX and communication-based methods.

该研究提出了一种新的训练方法，该方法在训练初期通过共享信息和重构信息来促进智能体之间的合作，随着训练的进行，逐渐过渡到完全去中心化的执行模式，实验结果表明该方法的性能不逊于基于传统通信的方法。

从显式通信到默示合作：一种新的协作多智能体强化学习范式

From Explicit Communication to Tacit Cooperation:A Novel Paradigm for  Cooperative MARL

Explicit communication among humans is key to coordinating and learning.
Social learning, which uses cues from experts, can greatly benefit from the
usage of explicit communication to align heterogeneous policies, reduce sample
complexity, and solve partially observable tasks. Emergent communication, a
type of explicit communication, studies the creation of an artificial language
to encode a high task-utility message directly from data. However, in most
cases, emergent communication sends insufficiently compressed messages with
little or null information, which also may not be understandable to a
third-party listener. This paper proposes an unsupervised method based on the
information bottleneck to capture both referential complexity and task-specific
utility to adequately explore sparse social communication scenarios in
multi-agent reinforcement learning (MARL). We show that our model is able to i)
develop a natural-language-inspired lexicon of messages that is independently
composed of a set of emergent concepts, which span the observations and intents
with minimal bits, ii) develop communication to align the action policies of
heterogeneous agents with dissimilar feature models, and iii) learn a
communication policy from watching an expert's action policy, which we term
`social shadowing'.

本文提出了一种基于信息瓶颈的无监督方法，用于探索非常稀疏的多智能体增强学习中的社交通信情境，该方法可以捕捉引用复杂性和任务特定效用，并开发出一种自然语言灵感的信息组成的词汇表，该词汇表独立于一组紧急概念，使其具有极小的位数，同时可以使用社交影子的观测构成，并通过社交影子来学习通信策略。