Multi-Agent Reinforcement Learning (MARL) algorithms face the challenge of
efficient exploration due to the exponential increase in the size of the joint
state-action space. While demonstration-guided learning has proven beneficial
in single-agent settings, its direct applicability to MARL is hindered by the
practical difficulty of obtaining joint expert demonstrations. In this work, we
introduce a novel concept of personalized expert demonstrations, tailored for
each individual agent or, more broadly, each individual type of agent within a
heterogeneous team. These demonstrations solely pertain to single-agent
behaviors and how each agent can achieve personal goals without encompassing
any cooperative elements, thus naively imitating them will not achieve
cooperation due to potential conflicts. To this end, we propose an approach
that selectively utilizes personalized expert demonstrations as guidance and
allows agents to learn to cooperate, namely personalized expert-guided MARL
(PegMARL). This algorithm utilizes two discriminators: the first provides
incentives based on the alignment of policy behavior with demonstrations, and
the second regulates incentives based on whether the behavior leads to the
desired objective. We evaluate PegMARL using personalized demonstrations in
both discrete and continuous environments. The results demonstrate that PegMARL
learns near-optimal policies even when provided with suboptimal demonstrations,
and outperforms state-of-the-art MARL algorithms in solving coordinated tasks.
We also showcase PegMARL's capability to leverage joint demonstrations in the
StarCraft scenario and converge effectively even with demonstrations from
non-co-trained policies.

在多智能体强化学习中，引入个性化专家演示，为每个个体智能体或团队中的每个个体智能体类型定制，以实现合作任务，并展示其在离散和连续环境中的性能优于现有算法。