Abstract:In recent years, research in the area of human-robot interaction has focused on developing robots capable of understanding complex human instructions and performing tasks in dynamic and diverse environments. These systems have a wide range of applications, from personal assistance to industrial robotics, emphasizing the importance of robots interacting flexibly, naturally and safely with humans. This paper presents an advanced architecture for robotic action planning that integrates communication, perception, and planning with Large Language Models (LLMs). Our system is designed to translate commands expressed in natural language into executable robot actions, incorporating environmental information and dynamically updating plans based on real-time feedback. The Planner Module is the core of the system where LLMs embedded in a modified ReAct framework are employed to interpret and carry out user commands. By leveraging their extensive pre-trained knowledge, LLMs can effectively process user requests without the need to introduce new knowledge on the changing environment. The modified ReAct framework further enhances the execution space by providing real-time environmental perception and the outcomes of physical actions. By combining robust and dynamic semantic map representations as graphs with control components and failure explanations, this architecture enhances a robot adaptability, task execution, and seamless collaboration with human users in shared and dynamic environments. Through the integration of continuous feedback loops with the environment the system can dynamically adjusts the plan to accommodate unexpected changes, optimizing the robot ability to perform tasks. Using a dataset of previous experience is possible to provide detailed feedback about the failure. Updating the LLMs context of the next iteration with suggestion on how to overcame the issue.

Language, Camera, Autonomy! Prompt-engineered Robot Control for Rapidly Evolving Deployment

NARRATE: Versatile Language Architecture for Optimal Control in Robotics

LLM Granularity for On-the-Fly Robot Control

MHRC: Closed-loop Decentralized Multi-Heterogeneous Robot Collaboration with Large Language Models

Deployment of Large Language Models to Control Mobile Robots at the Edge

REAL: Resilience and Adaptation using Large Language Models on Autonomous Aerial Robots

Enhancing the LLM-Based Robot Manipulation Through Human-Robot Collaboration

A Smart Interactive Camera Robot Based on Large Language Models

Prompt a Robot to Walk with Large Language Models

Lifelong Robot Library Learning: Bootstrapping Composable and Generalizable Skills for Embodied Control with Language Models

Hey Robot! Personalizing Robot Navigation through Model Predictive Control with a Large Language Model

Time is on my sight: scene graph filtering for dynamic environment perception in an LLM-driven robot

One to rule them all: natural language to bind communication, perception and action

Learning to Learn Faster from Human Feedback with Language Model Predictive Control

Language-Grounded Control for Coordinated Robot Motion and Speech

Lifelong Robot Learning with Human Assisted Language Planners

LIT: Large Language Model Driven Intention Tracking for Proactive Human-Robot Collaboration -- A Robot Sous-Chef Application

Autonomous Behavior Planning For Humanoid Loco-manipulation Through Grounded Language Model

Prompt, Plan, Perform: LLM-based Humanoid Control via Quantized Imitation Learning

LANCAR: Leveraging Language for Context-Aware Robot Locomotion in Unstructured Environments

Grounding Language Models in Autonomous Loco-manipulation Tasks