Abstract:In recent years, research in the area of human-robot interaction has focused on developing robots capable of understanding complex human instructions and performing tasks in dynamic and diverse environments. These systems have a wide range of applications, from personal assistance to industrial robotics, emphasizing the importance of robots interacting flexibly, naturally and safely with humans. This paper presents an advanced architecture for robotic action planning that integrates communication, perception, and planning with Large Language Models (LLMs). Our system is designed to translate commands expressed in natural language into executable robot actions, incorporating environmental information and dynamically updating plans based on real-time feedback. The Planner Module is the core of the system where LLMs embedded in a modified ReAct framework are employed to interpret and carry out user commands. By leveraging their extensive pre-trained knowledge, LLMs can effectively process user requests without the need to introduce new knowledge on the changing environment. The modified ReAct framework further enhances the execution space by providing real-time environmental perception and the outcomes of physical actions. By combining robust and dynamic semantic map representations as graphs with control components and failure explanations, this architecture enhances a robot adaptability, task execution, and seamless collaboration with human users in shared and dynamic environments. Through the integration of continuous feedback loops with the environment the system can dynamically adjusts the plan to accommodate unexpected changes, optimizing the robot ability to perform tasks. Using a dataset of previous experience is possible to provide detailed feedback about the failure. Updating the LLMs context of the next iteration with suggestion on how to overcame the issue.

Online Knowledge Acquisition And General Problem Solving In A Real World By Humanoid Robots

A service robot architecture supporting individualized coordination

Developing high-level cognitive functions for service robots

Knowledge Model-Based Adaptive Intelligent Control Of Robots For A Symbiotic Human-Robot System

Integrating Robot Task Planner with Common-sense Knowledge Base to Improve the Efficiency of Planning

An Epistemic Human-Aware Task Planner which Anticipates Human Beliefs and Decisions

DAC-h3: A Proactive Robot Cognitive Architecture to Acquire and Express Knowledge About the World and the Self

Interactive Continual Learning Architecture for Long-Term Personalization of Home Service Robots

Knowledge-Based Autonomous Behavior Control of Robots in A Symbiotic Human-Robot System

Toward Open Knowledge Enabling for Human-Robot Interaction

Remote Life Support Robot Interface System for Global Task Planning and Local Action Expansion Using Foundation Models

System for augmented human–robot interaction through mixed reality and robot training by non-experts in customer service environments

LAAIR: A Layered Architecture for Autonomous Interactive Robots

Human-Aware Robot Task Planning Based on a Hierarchical Task Model

Robot Safe Interaction System for Intelligent Industrial Co-Robots

Online Grounding of Symbolic Planning Domains in Unknown Environments

A Data-driven Approach Towards Human-robot Collaborative Problem Solving in a Shared Space

One to rule them all: natural language to bind communication, perception and action

How to Raise a Robot -- A Case for Neuro-Symbolic AI in Constrained Task Planning for Humanoid Assistive Robots

Robo-CSK-Organizer: Commonsense Knowledge to Organize Detected Objects for Multipurpose Robots

Developmental Cognitive Map Learning And Behavior Acquisition By A Mobile Robot Through A Self-Organizing Incremental Neural Network