Learning to cooperate in distributed partially observable environments with no communication abilities poses significant challenges for multi-agent deep reinforcement learning (MARL). This paper addresses key concerns in this domain, focusing on inferring state representations from individual agent observations and leveraging these representations to enhance agents' exploration and collaborative task execution policies. To this end, we propose a novel state modelling framework for cooperative MARL, where agents infer meaningful belief representations of the non-observable state, with respect to optimizing their own policies, while filtering redundant and less informative joint state information. Building upon this framework, we propose the MARL SMPE algorithm. In SMPE, agents enhance their own policy's discriminative abilities under partial observability, explicitly by incorporating their beliefs into the policy network, and implicitly by adopting an adversarial type of exploration policies which encourages agents to discover novel, high-value states while improving the discriminative abilities of others. Experimentally, we show that SMPE outperforms state-of-the-art MARL algorithms in complex fully cooperative tasks from the MPE, LBF, and RWARE benchmarks.

本研究解决了多智能体深度强化学习中，分布式部分可观察环境下缺乏通信能力所带来的合作学习挑战。提出了一种新颖的状态建模框架，帮助智能体通过推断非可观察状态的信念表征来优化其探索和合作执行策略。实验结果表明，所提出的MARL SMPE算法在多个复杂完全合作任务中超越了现有的最先进的MARL算法。

通过状态建模和对抗性探索增强合作多智能体强化学习