Abstract:With the increasing deployment of artificial intelligence (AI) technologies, the potential of humans working with AI agents has been growing at a great speed. Human-AI teaming is an important paradigm for studying various aspects when humans and AI agents work together. The unique aspect of Human-AI teaming research is the need to jointly study humans and AI agents, demanding multidisciplinary research efforts from machine learning to human-computer interaction, robotics, cognitive science, neuroscience, psychology, social science, and complex systems. However, existing platforms for Human-AI teaming research are limited, often supporting oversimplified scenarios and a single task, or specifically focusing on either human-teaming research or multi-agent AI algorithms. We introduce CREW, a platform to facilitate Human-AI teaming research and engage collaborations from multiple scientific disciplines, with a strong emphasis on human involvement. It includes pre-built tasks for cognitive studies and Human-AI teaming with expandable potentials from our modular design. Following conventional cognitive neuroscience research, CREW also supports multimodal human physiological signal recording for behavior analysis. Moreover, CREW benchmarks real-time human-guided reinforcement learning agents using state-of-the-art algorithms and well-tuned baselines. With CREW, we were able to conduct 50 human subject studies within a week to verify the effectiveness of our benchmark.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is the lack of a comprehensive and unified research platform in human - artificial intelligence (AI) collaborative research. Specifically, the existing human - AI team research platforms have the following limitations: 1. **Limited task diversity**: Most environments only support one type of task, and it is difficult to expand new tasks or modify existing tasks. 2. **Insufficient real - time interaction capabilities**: Existing solutions usually only support providing human feedback through the playback of offline trajectories, lacking a real - time feedback mechanism. 3. **Lack of multi - modal feedback and physiological data collection**: Existing platforms mainly focus on the collection of game data, such as states, actions or rewards, but do not support the collection and analysis of human physiological data (such as eye movements, pupil responses, brain activity, heart rate or natural language), which are crucial for understanding human cognitive states and different effects. 4. **Limited cooperation scale**: Most environments only support the interaction between two people in a cooperative or competitive setting. 5. **Small number of experimental participants**: Existing human - AI team research is often evaluated only among a small number of participants or authors, which greatly limits the understanding of state - of - the - art performance. To address these challenges, the paper introduces the CREW platform, aiming to promote human - AI team research and solve the above problems. The CREW platform has the following features: - **Scalable and open - environment design**: Provides built - in tasks for rapid development and allows users to integrate custom tasks. - **Real - time communication support**: Supports real - time communication between multiple humans and AI agents. - **Mixed human - AI team support**: Supports the cooperation and competition of multiple humans and multiple AI agents. - **Parallel session support**: Supports multiple independent parallel sessions, regardless of geographical location, to achieve the "crowdsourcing" effect for large - scale experiments. - **Comprehensive human data collection**: Provides interfaces to collect multi - modal human data simultaneously, including active instructions and feedback as well as passive physiological signals. - **Machine - learning - community - friendly algorithm design**: Selects programming languages and libraries to conform to the habits and preferences of the machine - learning community, and the system design is modular, facilitating seamless conversion between tasks and algorithms. Through these features, the CREW platform aims to become the infrastructure for multidisciplinary, reproducible and scalable human - AI team research.

CREW: Facilitating Human-AI Teaming Research

Human-AI Teamwork Interface Design Using Patterns of Interactions

AI Challenges in Human-Robot Cognitive Teaming

Enabling Multi-Robot Collaboration from Single-Human Guidance

Human-AI Collaboration in a Cooperative Game Setting

Human-AI Co-Learning for Data-Driven AI

AI-Empowered Human Research Integrating Brain Science and Social Sciences Insights

CoHRT: A Collaboration System for Human-Robot Teamwork

From Human-Human Collaboration to Human-AI Collaboration

Inner Attention Supported Adaptive Cooperation for Heterogeneous Multi Robots Teaming based on Multi-agent Reinforcement Learning

Open and Real-World Human-Ai Coordination by Heterogeneous Training with Communication

Effect of Adaptive Communication Support on Human-AI Collaboration

Human-Machine Teaming for UAVs: An Experimentation Platform

Language-Facilitated Human-Robot Cooperation within a Human Cognitive Modeling Infrastructure: A Case in Space Exploration Task

Human-AI Coordination via Human-Regularized Search and Learning

Demonstrating HumanTHOR: A Simulation Platform and Benchmark for Human-Robot Collaboration in a Shared Workspace

COMMUNICATION IN AI-ASSISTED TEAMS DURING AN INTERDISCIPLINARY DRONE DESIGN PROBLEM

Efficient Human-AI Coordination via Preparatory Language-based Convention

Designs for Enabling Collaboration in Human-Machine Teaming via Interactive and Explainable Systems

The Human Factor in AI Red Teaming: Perspectives from Social and Collaborative Computing

Real-World Human-Robot Collaborative Reinforcement Learning