Abstract:In recent years, research in the area of human-robot interaction has focused on developing robots capable of understanding complex human instructions and performing tasks in dynamic and diverse environments. These systems have a wide range of applications, from personal assistance to industrial robotics, emphasizing the importance of robots interacting flexibly, naturally and safely with humans. This paper presents an advanced architecture for robotic action planning that integrates communication, perception, and planning with Large Language Models (LLMs). Our system is designed to translate commands expressed in natural language into executable robot actions, incorporating environmental information and dynamically updating plans based on real-time feedback. The Planner Module is the core of the system where LLMs embedded in a modified ReAct framework are employed to interpret and carry out user commands. By leveraging their extensive pre-trained knowledge, LLMs can effectively process user requests without the need to introduce new knowledge on the changing environment. The modified ReAct framework further enhances the execution space by providing real-time environmental perception and the outcomes of physical actions. By combining robust and dynamic semantic map representations as graphs with control components and failure explanations, this architecture enhances a robot adaptability, task execution, and seamless collaboration with human users in shared and dynamic environments. Through the integration of continuous feedback loops with the environment the system can dynamically adjusts the plan to accommodate unexpected changes, optimizing the robot ability to perform tasks. Using a dataset of previous experience is possible to provide detailed feedback about the failure. Updating the LLMs context of the next iteration with suggestion on how to overcame the issue.

Interactive Robot Learning of Gestures, Language and Affordances

Learning Physical Collaborative Robot Behaviors From Human Demonstrations

Learning Interactive Affordance for Human-Robot Interaction

Affordance Learning And Inference Based On Vision-Speech Association In Human-Robot Interactions

Incremental Learning of Humanoid Robot Behavior from Natural Interaction and Large Language Models

Learning Social Affordance for Human-Robot Interaction

Simultaneously learning intentions and preferences during physical human-robot cooperation

Learning at the Ends: From Hand to Tool Affordances in Humanoid Robots

One to rule them all: natural language to bind communication, perception and action

Improving Human Legibility in Collaborative Robot Tasks through Augmented Reality and Workspace Preparation

Machines Like Us and People Like You: Toward Human-Robot Shared Experience

Unified Learning from Demonstrations, Corrections, and Preferences during Physical Human-Robot Interaction

Enhancing the LLM-Based Robot Manipulation Through Human-Robot Collaboration

Towards Abstract Relational Learning in Human Robot Interaction

$π_0$: A Vision-Language-Action Flow Model for General Robot Control

Enabling Multi-Robot Collaboration from Single-Human Guidance

Aligning Learning with Communication in Shared Autonomy

Enabling Intuitive Human-Robot Teaming Using Augmented Reality and Gesture Control

Learning Human-to-Robot Handovers from Point Clouds.

Affordance Perception by a Knowledge-Guided Vision-Language Model with Efficient Error Correction

Multi-Modal Human-Machine Communication for Instructing Robot Grasping Tasks