COHERENT: Collaboration of Heterogeneous Multi-Robot System with Large Language Models

Kehui Liu,Zixin Tang,Dong Wang,Zhigang Wang,Bin Zhao,Xuelong Li

2024-09-25

Abstract:Leveraging the powerful reasoning capabilities of large language models (LLMs), recent LLM-based robot task planning methods yield promising results. However, they mainly focus on single or multiple homogeneous robots on simple tasks. Practically, complex long-horizon tasks always require collaborations among multiple heterogeneous robots especially with more complex action spaces, which makes these tasks more challenging. To this end, we propose COHERENT, a novel LLM-based task planning framework for collaboration of heterogeneous multi-robot systems including quadrotors, robotic dogs, and robotic arms. Specifically, a Proposal-Execution-Feedback-Adjustment (PEFA) mechanism is designed to decompose and assign actions for individual robots, where a centralized task assigner makes a task planning proposal to decompose the complex task into subtasks, and then assigns subtasks to robot executors. Each robot executor selects a feasible action to implement the assigned subtask and reports self-reflection feedback to the task assigner for plan adjustment. The PEFA loops until the task is completed. Moreover, we create a challenging heterogeneous multi-robot task planning benchmark encompassing 100 complex long-horizon tasks. The experimental results show that our work surpasses the previous methods by a large margin in terms of success rate and execution efficiency. The experimental videos, code, and benchmark are released at <a class="link-external link-https" href="https://github.com/MrKeee/COHERENT" rel="external noopener nofollow">this https URL</a>.

Robotics,Artificial Intelligence

What problem does this paper attempt to address?

This paper aims to address the cooperation and task allocation issues in heterogeneous multi-robot systems when executing complex long-term tasks. Specifically, existing robot task planning methods based on large language models (LLMs) mainly focus on single or homogeneous multi-robot systems and usually handle relatively simple tasks. However, in practical applications, complex long-term tasks often require collaboration between different types of robots (with different action spaces), making the tasks more challenging. To solve this problem, the paper proposes the COHERENT framework, a novel LLMs-based heterogeneous multi-robot task planning framework. This framework decomposes and allocates tasks to various robots through the design of a Proposal-Execution-Feedback-Adjustment (PEFA) mechanism, thereby achieving effective planning and execution of complex tasks. Additionally, the authors created a benchmark environment containing 100 complex long-term tasks, and experimental results show that COHERENT significantly outperforms existing methods in terms of task success rate and execution efficiency.

COHERENT: Collaboration of Heterogeneous Multi-Robot System with Large Language Models

Leveraging Large Language Model for Heterogeneous Ad Hoc Teamwork Collaboration

Meeting-Merging-Mission: A Multi-robot Coordinate Framework for Large-Scale Communication-Limited Exploration.

MHRC: Closed-loop Decentralized Multi-Heterogeneous Robot Collaboration with Large Language Models

RoCo: Dialectic Multi-Robot Collaboration with Large Language Models

Nl2Hltl2Plan: Scaling Up Natural Language Understanding for Multi-Robots Through Hierarchical Temporal Logic Task Representation

LaMMA-P: Generalizable Multi-Agent Long-Horizon Task Allocation and Planning with LM-Driven PDDL Planner

FLTRNN: Faithful Long-Horizon Task Planning for Robotics with Large Language Models

$\textbf{EMOS}$: $\textbf{E}$mbodiment-aware Heterogeneous $\textbf{M}$ulti-robot $\textbf{O}$perating $\textbf{S}$ystem with LLM Agents

Leveraging the Efficiency of Multi-Task Robot Manipulation Via Task-Evoked Planner and Reinforcement Learning

Long-horizon Locomotion and Manipulation on a Quadrupedal Robot with Large Language Models

MLDT: Multi-Level Decomposition for Complex Long-Horizon Robotic Task Planning with Open-Source Large Language Model

Hierarchical Large Language Models in Cloud Edge End Architecture for Heterogeneous Robot Cluster Control

SELP: Generating Safe and Efficient Task Plans for Robot Agents with Large Language Models

Building Cooperative Embodied Agents Modularly with Large Language Models

Large Language Models for Orchestrating Bimanual Robots

CoAct: A Global-Local Hierarchy for Autonomous Agent Collaboration

Large Language Models for Robotics: Opportunities, Challenges, and Perspectives

Enhancing Robot Task Planning and Execution through Multi-Layer Large Language Models

SMART-LLM: Smart Multi-Agent Robot Task Planning using Large Language Models