Abstract:The integration of large language models (LLMs) with robotics has significantly advanced robots' abilities in perception, cognition, and task planning. The use of natural language interfaces offers a unified approach for expressing the capability differences of heterogeneous robots, facilitating communication between them, and enabling seamless task allocation and collaboration. Currently, the utilization of LLMs to achieve decentralized multi-heterogeneous robot collaborative tasks remains an under-explored area of research. In this paper, we introduce a novel framework that utilizes LLMs to achieve decentralized collaboration among multiple heterogeneous robots. Our framework supports three robot categories, mobile robots, manipulation robots, and mobile manipulation robots, working together to complete tasks such as exploration, transportation, and organization. We developed a rich set of textual feedback mechanisms and chain-of-thought (CoT) prompts to enhance task planning efficiency and overall system performance. The mobile manipulation robot can adjust its base position flexibly, ensuring optimal conditions for grasping tasks. The manipulation robot can comprehend task requirements, seek assistance when necessary, and handle objects appropriately. Meanwhile, the mobile robot can explore the environment extensively, map object locations, and communicate this information to the mobile manipulation robot, thus improving task execution efficiency. We evaluated the framework using PyBullet, creating scenarios with three different room layouts and three distinct operational tasks. We tested various LLM models and conducted ablation studies to assess the contributions of different modules. The experimental results confirm the effectiveness and necessity of our proposed framework.

What problem does this paper attempt to address?

### What problems does this paper attempt to solve? This paper aims to address the challenges of multi - heterogeneous robots in decentralized collaborative tasks. Specifically, the authors propose a new framework named MHRC (Multi - Heterogeneous Robot Collaboration), which utilizes large language models (LLMs) to achieve decentralized collaboration among multiple heterogeneous robots. The following are the main problems and goals of this research: 1. **Collaboration challenges of multi - heterogeneous robots**: - In the real world, a single robot is difficult to handle all aspects of complex tasks, so multiple robots with different capabilities are required to work together. - Traditional centralized control methods are rigid and inefficient in managing complex scenarios. 2. **Application of natural language interfaces**: - Use natural language interfaces to uniformly express the capability differences among different robots, promote communication between them, and achieve seamless task allocation and collaboration. - The significant progress of large language models in natural language understanding, dialogue generation, extensive world knowledge, and complex reasoning provides new opportunities for robot collaboration. 3. **Specific tasks and environmental simulation**: - The research designs three specific tasks: Pack Objects, Sort Solids, and Make Sandwich, to evaluate the robots' capabilities in basic operations, color matching, and sequential stacking, etc. - Use the PyBullet simulation platform to create an experimental environment containing different room layouts and furniture arrangements to simulate real - life scenarios. 4. **Key modules of the framework**: - **Observation Module**: Collect all necessary state information for task and motion planning, including the scene graph of the environment, messages between robots, and the state information of individual robots. - **Memory Module**: Record the feedback history, the history of received messages, and the action history to ensure that the context remains consistent in long - sequence tasks. - **Planning Module**: Utilize large language models to generate actions and perform complex task planning and decision - making through the Chain - of - Thought (CoT) framework. 5. **Verification and evaluation**: - Verify the effectiveness of this framework through experiments, demonstrating high success rates and efficient collaboration capabilities. - Emphasize the flexibility and autonomy of large language models in decentralized multi - heterogeneous robot collaboration. Each robot can make decisions independently based on its own observations and communications without relying on a central controller. In summary, this paper attempts to improve the performance of multi - heterogeneous robots in decentralized collaborative tasks by introducing large language models, thereby increasing the success rate and efficiency of task execution.

MHRC: Closed-loop Decentralized Multi-Heterogeneous Robot Collaboration with Large Language Models

Leveraging Large Language Model for Heterogeneous Ad Hoc Teamwork Collaboration

Large Language Models for Robotics: Opportunities, Challenges, and Perspectives

Enhancing the LLM-Based Robot Manipulation Through Human-Robot Collaboration

COHERENT: Collaboration of Heterogeneous Multi-Robot System with Large Language Models

When Robots Get Chatty: Grounding Multimodal Human-Robot Conversation and Collaboration

Decision-Making in Robotic Grasping with Large Language Models.

Automatic Robotic Development through Collaborative Framework by Large Language Models

Enhancing Robot Task Planning and Execution through Multi-Layer Large Language Models

LaMI: Large Language Models for Multi-Modal Human-Robot Interaction

Hierarchical Large Language Models in Cloud Edge End Architecture for Heterogeneous Robot Cluster Control

Conversational Language Models for Human-in-the-Loop Multi-Robot Coordination

Long-horizon Locomotion and Manipulation on a Quadrupedal Robot with Large Language Models

RoCo: Dialectic Multi-Robot Collaboration with Large Language Models

Nl2Hltl2Plan: Scaling Up Natural Language Understanding for Multi-Robots Through Hierarchical Temporal Logic Task Representation

Large Language Models for Orchestrating Bimanual Robots

Grounding Language Models in Autonomous Loco-manipulation Tasks

LLM-Based Human-Robot Collaboration Framework for Manipulation Tasks

Understanding Large-Language Model (LLM)-powered Human-Robot Interaction

A Smart Interactive Camera Robot Based on Large Language Models