Abstract:Cooperative Multi-agent Reinforcement Learning (MARL) algorithms with Zero-Shot Coordination (ZSC) have gained significant attention in recent years. ZSC refers to the ability of agents to coordinate zero-shot (without additional interaction experience) with independently trained agents. While ZSC is crucial for cooperative MARL agents, it might not be possible for complex tasks and changing environments. Agents also need to adapt and improve their performance with minimal interaction with other agents. In this work, we show empirically that state-of-the-art ZSC algorithms have poor performance when paired with agents trained with different learning methods, and they require millions of interaction samples to adapt to these new partners. To investigate this issue, we formally defined a framework based on a popular cooperative multi-agent game called Hanabi to evaluate the adaptability of MARL methods. In particular, we created a diverse set of pre-trained agents and defined a new metric called adaptation regret that measures the agent's ability to efficiently adapt and improve its coordination performance when paired with some held-out pool of partners on top of its ZSC performance. After evaluating several SOTA algorithms using our framework, our experiments reveal that naive Independent Q-Learning (IQL) agents in most cases adapt as quickly as the SOTA ZSC algorithm Off-Belief Learning (OBL). This finding raises an interesting research question: How to design MARL algorithms with high ZSC performance and capability of fast adaptation to unseen partners. As a first step, we studied the role of different hyper-parameters and design choices on the adaptability of current MARL algorithms. Our experiments show that two categories of hyper-parameters controlling the training data diversity and optimization process have a significant impact on the adaptability of Hanabi agents.

Multi-Expert Distillation for Few-Shot Coordination (student Abstract)

Bi-Level Orthogonal Multi-Teacher Distillation

Coordination Scheme Probing for Generalizable Multi-Agent Reinforcement Learning

Maximum Entropy Population Based Training for Zero-Shot Human-AI Coordination

Cooperative multi-agent target searching: a deep reinforcement learning approach based on parallel hindsight experience replay

Learning Distilled Collaboration Graph for Multi-Agent Perception

Online Knowledge Distillation with Diverse Peers

Dual Policy Distillation

Better Together: Data-Free Multi-Student Coevolved Distillation

Learning Multi-Agent Cooperation via Considering Actions of Teammates

Adaptive Multi-Teacher Knowledge Distillation with Meta-Learning

Transfer Heterogeneous Knowledge Among Peer-to-Peer Teammates: A Model Distillation Approach

Discovering Generalizable Multi-agent Coordination Skills from Multi-task Offline Data

Teaching-Assistant-in-the-Loop: Improving Knowledge Distillation from Imperfect Teacher Models in Low-Budget Scenarios

Reinforced Multi-Teacher Selection for Knowledge Distillation

Reprogramming Distillation for Medical Foundation Models

Multi-Level Feature Distillation of Joint Teachers Trained on Distinct Image Datasets

Towards Few-shot Coordination: Revisiting Ad-hoc Teamplay Challenge In the Game of Hanabi

Coach-Player Multi-Agent Reinforcement Learning for Dynamic Team Composition

AgentCoord: Visually Exploring Coordination Strategy for LLM-based Multi-Agent Collaboration

Dual teachers for self-knowledge distillation