Abstract:In recent years, research in the area of human-robot interaction has focused on developing robots capable of understanding complex human instructions and performing tasks in dynamic and diverse environments. These systems have a wide range of applications, from personal assistance to industrial robotics, emphasizing the importance of robots interacting flexibly, naturally and safely with humans. This paper presents an advanced architecture for robotic action planning that integrates communication, perception, and planning with Large Language Models (LLMs). Our system is designed to translate commands expressed in natural language into executable robot actions, incorporating environmental information and dynamically updating plans based on real-time feedback. The Planner Module is the core of the system where LLMs embedded in a modified ReAct framework are employed to interpret and carry out user commands. By leveraging their extensive pre-trained knowledge, LLMs can effectively process user requests without the need to introduce new knowledge on the changing environment. The modified ReAct framework further enhances the execution space by providing real-time environmental perception and the outcomes of physical actions. By combining robust and dynamic semantic map representations as graphs with control components and failure explanations, this architecture enhances a robot adaptability, task execution, and seamless collaboration with human users in shared and dynamic environments. Through the integration of continuous feedback loops with the environment the system can dynamically adjusts the plan to accommodate unexpected changes, optimizing the robot ability to perform tasks. Using a dataset of previous experience is possible to provide detailed feedback about the failure. Updating the LLMs context of the next iteration with suggestion on how to overcame the issue.

From Natural Language to Long-Range Path Plans in Outdoor Environments

Using Language to Generate State Abstractions for Long-Range Planning in Outdoor Environments

SayPlan: Grounding Large Language Models using 3D Scene Graphs for Scalable Task Planning

SayPlan: Grounding Large Language Models using 3D Scene Graphs for Scalable Robot Task Planning

Reasoning about the Unseen for Efficient Outdoor Object Navigation

Learning Models for Following Natural Language Directions in Unknown Environments

Online Grounding of Symbolic Planning Domains in Unknown Environments

3P-LLM: Probabilistic Path Planning using Large Language Model for Autonomous Robot Navigation

Grounding Language Models in Autonomous Loco-manipulation Tasks

Language-Augmented Symbolic Planner for Open-World Task Planning

Leveraging Environment Interaction for Automated PDDL Translation and Planning with Large Language Models

Autonomous Behavior Planning For Humanoid Loco-manipulation Through Grounded Language Model

Language-guided Robust Navigation for Mobile Robots in Dynamically-changing Environments

On Grounded Planning for Embodied Tasks with Language Models

Prospection: Interpretable Plans From Language By Predicting the Future

Plan-Seq-Learn: Language Model Guided RL for Solving Long Horizon Robotics Tasks

Translating Natural Language to Planning Goals with Large-Language Models

Open-vocabulary Queryable Scene Representations for Real World Planning

SELP: Generating Safe and Efficient Task Plans for Robot Agents with Large Language Models

PlanAgent: A Multi-modal Large Language Agent for Closed-loop Vehicle Motion Planning

One to rule them all: natural language to bind communication, perception and action