Abstract:In human-robots collaboration scenarios, a human would give robots an instruction that is intuitive for the human himself to accomplish. However, the instruction given to robots is likely ambiguous for them to understand as some information is implicit in the instruction. Therefore, it is necessary for the robots to jointly reason the operation details and perform the embodied multi-agent task planning given the ambiguous instruction. This problem exhibits significant challenges in both language understanding and dynamic task planning with the perception information. In this work, an embodied multi-agent task planning framework is proposed to utilize external knowledge sources and dynamically perceived visual information to resolve the high-level instructions, and dynamically allocate the decomposed tasks to multiple agents. Furthermore, we utilize the semantic information to perform environment perception and generate sub-goals to achieve the navigation motion. This model effectively bridges the difference between the simulation environment and the physical environment, thus it can be simultaneously applied in both simulation and physical scenarios and avoid the notorious sim2real problem. Finally, we build a benchmark dataset to validate the embodied multi-agent task planning problem, which includes three types of high-level instructions in which some target objects are implicit in instructions. We perform the evaluation experiments on the simulation platform and in physical scenarios, demonstrating that the proposed model can achieve promising results for multi-agent collaborative tasks.

Semantic-based Dynamic Positioning Mechanism for Problem Solving in Multi-Agent Systems

Implementation of Multi-agent Based Hybrid Architecture for Autonomous Mobile Robots

Semantic mapping with multi-agent systems

Active semantic localization of mobile robot using partial observable Monte Carlo Planning

Cooperative Problem Solving Process Based on MAS_NP Model

Ontology Based Agent Services Compatible Matchmaking Mechanism

A MULTI-AGENT BASED COOPERATIVE INTELLIGENT FRAME FOR PROBLEM SOLVING METHOD

Multi-Agent Path Finding with Heterogeneous Geometric and Kinematic Constraints in Continuous Space

Autonomous Search of Semantic Objects in Unknown Environments

A Study On Service-Based Multi-Agent Geospatial Data Sharing Method

A Dynamic Model of Multi-Agent System

Hybrid Decision Making for Scalable Multi-Agent Navigation: Integrating Semantic Maps, Discrete Coordination, and Model Predictive Control

Embodied Multi-Agent Task Planning from Ambiguous Instruction

Semantic Grounding for Long-Term Autonomy of Mobile Robots Towards Dynamic Object Search in Home Environments

Automatic Configuration of Multi-Agent Model Predictive Controllers based on Semantic Graph World Models

Exploring Semantic Technologies in Service Matchmakin

A Self-organisation Model for Mobile Robots in Large Structure Assembly Using Multi-agent Systems

Dynamic Interaction Protocol Load in Multi-Agent System Collaboration

Summary of Research on Satellite Mission Planning Based on Multi-Agent-System

Agent-Oriented Planning in Multi-Agent Systems

Communication Based Multi-Agent Automated Negotiation in MAS