Abstract:Natural-language dialog is key for intuitive human-robot interaction. It can be used not only to express humans' intents, but also to communicate instructions for improvement if a robot does not understand a command correctly. Of great importance is to endow robots with the ability to learn from such interaction experience in an incremental way to allow them to improve their behaviors or avoid mistakes in the future. In this paper, we propose a system to achieve incremental learning of complex behavior from natural interaction, and demonstrate its implementation on a humanoid robot. Building on recent advances, we present a system that deploys Large Language Models (LLMs) for high-level orchestration of the robot's behavior, based on the idea of enabling the LLM to generate Python statements in an interactive console to invoke both robot perception and action. The interaction loop is closed by feeding back human instructions, environment observations, and execution results to the LLM, thus informing the generation of the next statement. Specifically, we introduce incremental prompt learning, which enables the system to interactively learn from its mistakes. For that purpose, the LLM can call another LLM responsible for code-level improvements of the current interaction based on human feedback. The improved interaction is then saved in the robot's memory, and thus retrieved on similar requests. We integrate the system in the robot cognitive architecture of the humanoid robot ARMAR-6 and evaluate our methods both quantitatively (in simulation) and qualitatively (in simulation and real-world) by demonstrating generalized incrementally-learned knowledge.

Training an Interactive Humanoid Robot Using Multimodal Deep Reinforcement Learning

A Data-Efficient Deep Learning Approach for Deployable Multimodal Social Robots

Multimodal Reinforcement Learning for Robots Collaborating with Humans

Incremental Learning of Humanoid Robot Behavior from Natural Interaction and Large Language Models

Robot gains Social Intelligence through Multimodal Deep Reinforcement Learning

Real-World Human-Robot Collaborative Reinforcement Learning

Multimodal integration learning of robot behavior using deep neural networks

Bootstrapping Adaptive Human-Machine Interfaces with Offline Reinforcement Learning

An End-to-End Human Simulator for Task-Oriented Multimodal Human-Robot Collaboration

Learning Agile Soccer Skills for a Bipedal Robot with Deep Reinforcement Learning

Multimodal Human-Autonomous Agents Interaction Using Pre-Trained Language and Visual Foundation Models

Multimodal Interactive Learning of Primitive Actions

An adaptive reinforcement learning-based multimodal data fusion framework for human-robot confrontation gaming

Words into Action: Learning Diverse Humanoid Robot Behaviors using Language Guided Iterative Motion Refinement

Learning Robot Soccer from Egocentric Vision with Deep Reinforcement Learning

Multimodal representation models for prediction and control from partial information

Learning Multimodal Latent Dynamics for Human-Robot Interaction

Multi-Modal Human-Machine Communication for Instructing Robot Grasping Tasks

Seamless Integration and Coordination of Cognitive Skills in Humanoid Robots: A Deep Learning Approach

Deep Learning-based Multimodal Control Interface for Human-Robot Collaboration

Reinforcement Learning based Embodied Agents Modelling Human Users Through Interaction and Multi-Sensory Perception