Abstract:This paper presents a novel approach to multi-robot planning and collaboration. We demonstrate a cognitive strategy for robots in human-robot teams that incorporates metacognition, natural language communication, and explainability. The system is embodied using the HARMONIC architecture that flexibly integrates cognitive and control capabilities across the team. We evaluate our approach through simulation experiments involving a joint search task by a team of heterogeneous robots (a UGV and a drone) and a human. We detail the system's handling of complex, real-world scenarios, effective action coordination between robots with different capabilities, and natural human-robot communication. This work demonstrates that the robots' ability to reason about plans, goals, and attitudes, and to provide explanations for actions and decisions are essential prerequisites for realistic human-robot teaming.

What problem does this paper attempt to address?

### Problems the Paper Attempts to Solve This paper aims to address a series of cognitive and control challenges faced by multi-robot teams when collaborating with humans. Specifically, the paper focuses on the following aspects: 1. **Understanding of Team Tasks and Organization**: - Robots need to be able to understand the team's tasks and organizational structure, including the responsibilities, preferences, and actions of each member. 2. **Development and Enhancement of Specific Capabilities**: - Robots need to develop and enhance specific capabilities, such as understanding the responsibilities, preferences, and actions of themselves and other team members. 3. **Maintenance and Use of Situational Memory**: - Robots need to maintain and use situational memory of past activities to make informed predictions and decisions based on previous experiences. 4. **Interpretation of Language, Visual, and Tactile Perception**: - Robots need to be able to interpret language, visual, and tactile perceptions into formal representations in a world model (ontology). 5. **Natural Language Communication**: - Robots need to be able to use natural language to communicate meaningfully with teammates. The realization of these capabilities will facilitate complex team interactions, attention management, goal setting, planning (including the integration of multiple multi-agent planning strategies), and the ability to handle unexpected challenges. Through these capabilities, robots can collaborate more effectively with humans, enhancing transparency and explainability, thereby building trust. The paper proposes a new multi-robot planning and collaboration method, which flexibly integrates cognitive and control capabilities into the team through the HARMONIC architecture. This architecture achieves bidirectional communication between the strategic cognitive layer and the tactical control layer, supporting complex team task handling, effective action coordination, and natural human-robot communication. Through simulation experiments, the paper demonstrates the effectiveness of this system in handling complex real-world scenarios.

HARMONIC: Cognitive and Control Collaboration in Human-Robotic Teams

HARMONIC: A Framework for Explanatory Cognitive Robots

CoHRT: A Collaboration System for Human-Robot Teamwork

A Modular Framework for Flexible Planning in Human-Robot Collaboration

Embodied, Intelligent Communication for Multi-Agent Cooperation

HARMONIC: A Multimodal Dataset of Assistive Human-Robot Collaboration

HARMONIOUS -- Human-like reactive motion control and multimodal perception for humanoid robots

How to be Helpful? Supportive Behaviors and Personalization for Human-Robot Collaboration

Proactive Robot Control for Collaborative Manipulation Using Human Intent

HARPS: An Online POMDP Framework for Human-Assisted Robotic Planning and Sensing

Collaborative Behavior Models for Optimized Human-Robot Teamwork

ARMCHAIR: integrated inverse reinforcement learning and model predictive control for human-robot collaboration

Enabling Intuitive Human-Robot Teaming Using Augmented Reality and Gesture Control

An Augmented Discrete-Time Approach for Human-Robot Collaboration

Designs for Enabling Collaboration in Human-Machine Teaming via Interactive and Explainable Systems

Enabling Human-Robot Collaboration via Holistic Human Perception and Partner-Aware Control

Collaborative Interaction Models for Optimized Human-Robot Teamwork.

Language-Facilitated Human-Robot Cooperation within a Human Cognitive Modeling Infrastructure: A Case in Space Exploration Task

Towards Efficient Human-Robot Collaboration With Robust Plan Recognition and Trajectory Prediction

Human Robot Collaboration: an Augmented Reality Approach—A Literature Review and Analysis

An Epistemic Human-Aware Task Planner which Anticipates Human Beliefs and Decisions