Abstract:Continual and interactive robot learning is a challenging problem as the robot is present with human users who expect the robot to learn novel skills to solve novel tasks perpetually with sample efficiency. In this work we present a framework for robots to query and learn visuo-motor robot skills and task relevant information via natural language dialog interactions with human users. Previous approaches either focus on improving the performance of instruction following agents, or passively learn novel skills or concepts. Instead, we used dialog combined with a language-skill grounding embedding to query or confirm skills and/or tasks requested by a user. To achieve this goal, we developed and integrated three different components for our agent. Firstly, we propose a novel visual-motor control policy ACT with Low Rank Adaptation (ACT-LoRA), which enables the existing SoTA ACT model to perform few-shot continual learning. Secondly, we develop an alignment model that projects demonstrations across skill embodiments into a shared embedding allowing us to know when to ask questions and/or demonstrations from users. Finally, we integrated an existing LLM to interact with a human user to perform grounded interactive continual skill learning to solve a task. Our ACT-LoRA model learns novel fine-tuned skills with a 100% accuracy when trained with only five demonstrations for a novel skill while still maintaining a 74.75% accuracy on pre-trained skills in the RLBench dataset where other models fall significantly short. We also performed a human-subjects study with 8 subjects to demonstrate the continual learning capabilities of our combined framework. We achieve a success rate of 75% in the task of sandwich making with the real robot learning from participant data demonstrating that robots can learn novel skills or task knowledge from dialogue with non-expert users using our approach.

CLFR-M: Continual Learning Framework for Robots Via Human Feedback and Dynamic Memory

Continual Learning through Human-Robot Interaction -- Human Perceptions of a Continual Learning Robot in Repeated Interactions

Interactive Continual Learning: Fast and Slow Thinking

FLTRNN: Faithful Long-Horizon Task Planning for Robotics with Large Language Models

Continual Learning for Autonomous Robots: A Prototype-based Approach

KlF: Knowledge Localization and Fusion for Language Model Continual Learning

Generalized Robot Learning Framework

LLM as A Robotic Brain: Unifying Egocentric Memory and Control

Continual Skill and Task Learning via Dialogue

MHRC: Closed-loop Decentralized Multi-Heterogeneous Robot Collaboration with Large Language Models

Online Active Continual Learning for Robotic Lifelong Object Recognition

Lifelong Robot Library Learning: Bootstrapping Composable and Generalizable Skills for Embodied Control with Language Models

Interactive Robot Learning from Verbal Correction

A Review of Continual Learning for Robotics

RAG-Modulo: Solving Sequential Tasks using Experience, Critics, and Language Models

A Novel Continuous Learning and Collaborative Decision Making Mechanism for Real-Time Cooperation of Humanoid Service Robots

A Modular Framework for Robot Embodied Instruction Following by Large Language Model

Incremental Learning of Humanoid Robot Behavior from Natural Interaction and Large Language Models

C3F: Constant Collaboration and Communication Framework for Graph-Representation Dynamic Multi-Robotic Systems

Interactive Continual Learning Architecture for Long-Term Personalization of Home Service Robots

Accelerating Reinforcement Learning of Robotic Manipulations via Feedback from Large Language Models