Abstract:Securing coordination between AI agent and teammates (human players or AI agents) in contexts involving unfamiliar humans continues to pose a significant challenge in Zero-Shot Coordination. The issue of cooperative incompatibility becomes particularly prominent when an AI agent is unsuccessful in synchronizing with certain previously unknown partners. Traditional algorithms have aimed to collaborate with partners by optimizing fixed objectives within a population, fostering diversity in strategies and behaviors. However, these techniques may lead to learning loss and an inability to cooperate with specific strategies within the population, a phenomenon named cooperative incompatibility in learning. In order to solve cooperative incompatibility in learning and effectively address the problem in the context of ZSC, we introduce the Cooperative Open-ended LEarning (COLE) framework, which formulates open-ended objectives in cooperative games with two players using perspectives of graph theory to evaluate and pinpoint the cooperative capacity of each strategy. We present two practical algorithms, specifically \algo and \algoR, which incorporate insights from game theory and graph theory. We also show that COLE could effectively overcome the cooperative incompatibility from theoretical and empirical analysis. Subsequently, we created an online Overcooked human-AI experiment platform, the COLE platform, which enables easy customization of questionnaires, model weights, and other aspects. Utilizing the COLE platform, we enlist 130 participants for human experiments. Our findings reveal a preference for our approach over state-of-the-art methods using a variety of subjective metrics. Moreover, objective experimental outcomes in the Overcooked game environment indicate that our method surpasses existing ones when coordinating with previously unencountered AI agents and the human proxy model.

Collaborating with Humans without Human Data

Learning to Cooperate with Humans using Generative Agents

Maximum Entropy Population Based Training for Zero-Shot Human-AI Coordination

Human-AI Coordination via Human-Regularized Search and Learning

Real-World Human-Robot Collaborative Reinforcement Learning

Reinforcement Learning on Human Decision Models for Uniquely Collaborative AI Teammates

Learning Zero-Shot Cooperation with Humans, Assuming Humans Are Biased

Diverse Conventions for Human-AI Collaboration

Asynchronous Training of Mixed-Role Human Actors in a Partially-Observable Environment

Collaboration of AI Agents via Cooperative Multi-Agent Deep Reinforcement Learning

Human-AI Collaboration in a Cooperative Game Setting

Collaborative Gym: A Framework for Enabling and Evaluating Human-Agent Collaboration

Multi-agent Collaboration for Feasible Collaborative Behavior Construction and Evaluation

Human-compatible driving partners through data-regularized self-play reinforcement learning

Two Approaches to Building Collaborative, Task-Oriented Dialog Agents through Self-Play

Enabling Multi-Robot Collaboration from Single-Human Guidance

Collaborative training of heterogeneous reinforcement learning agents in environments with sparse rewards: what and when to share?

Designs for Enabling Collaboration in Human-Machine Teaming via Interactive and Explainable Systems

Tackling Cooperative Incompatibility for Zero-Shot Human-AI Coordination

Social Interaction for Efficient Agent Learning from Human Reward.

Any-Play: An Intrinsic Augmentation for Zero-Shot Coordination