Abstract:In recent years, research in the area of human-robot interaction has focused on developing robots capable of understanding complex human instructions and performing tasks in dynamic and diverse environments. These systems have a wide range of applications, from personal assistance to industrial robotics, emphasizing the importance of robots interacting flexibly, naturally and safely with humans. This paper presents an advanced architecture for robotic action planning that integrates communication, perception, and planning with Large Language Models (LLMs). Our system is designed to translate commands expressed in natural language into executable robot actions, incorporating environmental information and dynamically updating plans based on real-time feedback. The Planner Module is the core of the system where LLMs embedded in a modified ReAct framework are employed to interpret and carry out user commands. By leveraging their extensive pre-trained knowledge, LLMs can effectively process user requests without the need to introduce new knowledge on the changing environment. The modified ReAct framework further enhances the execution space by providing real-time environmental perception and the outcomes of physical actions. By combining robust and dynamic semantic map representations as graphs with control components and failure explanations, this architecture enhances a robot adaptability, task execution, and seamless collaboration with human users in shared and dynamic environments. Through the integration of continuous feedback loops with the environment the system can dynamically adjusts the plan to accommodate unexpected changes, optimizing the robot ability to perform tasks. Using a dataset of previous experience is possible to provide detailed feedback about the failure. Updating the LLMs context of the next iteration with suggestion on how to overcame the issue.

Human Planning of Robot Actions through LLM-guided State Machine Synthesis

One to rule them all: natural language to bind communication, perception and action

Language models are robotic planners: reframing plans as goal refinement graphs

Autonomous Behavior Planning For Humanoid Loco-manipulation Through Grounded Language Model

Grounding LLMs For Robot Task Planning Using Closed-loop State Feedback

LLM-based Robot Task Planning with Exceptional Handling for General Purpose Service Robots

Natural Language as Policies: Reasoning for Coordinate-Level Embodied Control with LLMs

Verifiably Following Complex Robot Instructions with Foundation Models

Lifelong Robot Learning with Human Assisted Language Planners

LLM as BT-Planner: Leveraging LLMs for Behavior Tree Generation in Robot Task Planning

Towards Human Awareness in Robot Task Planning with Large Language Models

On the Exploration of LM-Based Soft Modular Robot Design

GG-LLM: Geometrically Grounding Large Language Models for Zero-shot Human Activity Forecasting in Human-Aware Task Planning

PLATO: Planning with LLMs and Affordances for Tool Manipulation

LAP, Using Action Feasibility for Improved Uncertainty Alignment of Large Language Model Planners

SELP: Generating Safe and Efficient Task Plans for Robot Agents with Large Language Models

Prompt, Plan, Perform: LLM-based Humanoid Control via Quantized Imitation Learning

Plug in the Safety Chip: Enforcing Constraints for LLM-driven Robot Agents

LLCoach: Generating Robot Soccer Plans using Multi-Role Large Language Models

Behavior Tree Generation using Large Language Models for Sequential Manipulation Planning with Human Instructions and Feedback