Cooperative Multi-agent Reinforcement Learning (MARL) algorithms with Zero-Shot Coordination (ZSC) have gained significant attention in recent years. ZSC refers to the ability of agents to coordinate zero-shot (without additional interaction experience) with independently trained agents. While ZSC is crucial for cooperative MARL agents, it might not be possible for complex tasks and changing environments. Agents also need to adapt and improve their performance with minimal interaction with other agents. In this work, we show empirically that state-of-the-art ZSC algorithms have poor performance when paired with agents trained with different learning methods, and they require millions of interaction samples to adapt to these new partners. To investigate this issue, we formally defined a framework based on a popular cooperative multi-agent game called Hanabi to evaluate the adaptability of MARL methods. In particular, we created a diverse set of pre-trained agents and defined a new metric called adaptation regret that measures the agent's ability to efficiently adapt and improve its coordination performance when paired with some held-out pool of partners on top of its ZSC performance. After evaluating several SOTA algorithms using our framework, our experiments reveal that naive Independent Q-Learning (IQL) agents in most cases adapt as quickly as the SOTA ZSC algorithm Off-Belief Learning (OBL). This finding raises an interesting research question: How to design MARL algorithms with high ZSC performance and capability of fast adaptation to unseen partners. As a first step, we studied the role of different hyper-parameters and design choices on the adaptability of current MARL algorithms. Our experiments show that two categories of hyper-parameters controlling the training data diversity and optimization process have a significant impact on the adaptability of Hanabi agents.

合作式多智能体强化学习与零样本协同在近年来引起了极大的关注。本文通过构建一个基于合作多智能体游戏Hanabi的框架，在使用最新零样本协同算法时探究了多智能体强化学习方法的适应性，并发现在大多数情况下，顺序独立强化学习代理（IQL）与最新的零样本协同算法Off-Belief Learning（OBL）相比，在适应性上表现相似。这一发现引发了一个有趣的研究问题：如何设计具有高零样本协同性能和快速适应未知伙伴的多智能体强化学习算法。通过研究不同超参数和设计选择对当前多智能体强化学习算法的适应性的作用，我们的实验表明，控制训练数据多样性和优化过程的两类超参数对Hanabi代理的适应性具有重要影响。

走向小样本协同：重新审视《汉尼拔》游戏中的即兴团队配合挑战