Abstract:Visual imitation learning is a promising approach that promotes robots to learn skills from visual demonstrations. However, current visual imitation learning approaches introduce unreasonable assumptions that the contexts of the visual demonstrations and the robot observations are consistent, which affects the flexibility and scalability of the approaches. It is a key challenge for robots to learn from visual demonstrations with inconsistent contexts. Inconsistent contexts may cause a serious difference in the pixel distribution of the operator and the environment, which makes vision-based control policies hardly effective. In this paper, we propose a novel imitation learning framework to enable robots to reproduce behavior by watching human demonstrations with inconsistent contexts, such as different viewpoints, operators, backgrounds, object appearances and positions. Specifically, our framework consists of three networks: flow-based viewpoint transformation network (FVTrans), robot2human alignment network (RANet) and inverse dynamics network (IDNet). First, FVTrans transforms various third-person demonstrations into the fixed robot execution view. With a meta learning strategy, FVTrans can quickly adapt to novel contexts with few samples. Then, RANet aligns the human and the robot at the feature level. Therefore, the demonstration feature can be used as a subgoal of the current moment. Finally, IDNet predicts the joint angles of the robot. We collect a multi-context dataset on the real robot (UR5) for three tasks, including grasping cups, sweeping garbage and placing objects. We empirically demonstrate that our framework can perform three tasks with a high success rate and be effectively generalized to different contexts.

Incremental learning for robot perception through HRI

Interactive Visual Task Learning for Robots

Incremental learning with neural networks for computer vision: a survey

Incremental procedural and sensorimotor learning in cognitive humanoid robots

Adaptive Incremental Learning of Image Semantics with Application to Social Robot

Incremental Learning Without Looking Back: a Neural Connection Relocation Approach

Incremental learning of high-level concepts by imitation

An Autonomous Developmental Cognitive Architecture Based on Incremental Associative Neural Network with Dynamic Audiovisual Fusion

Robot Learning from Human Demonstrations with Inconsistent Contexts

Robot Perception enables Complex Navigation Behavior via Self-Supervised Learning

Efficient Robot Learning for Perception and Mapping

3D_DEN: Open-ended 3D Object Recognition using Dynamically Expandable Networks

Incremental Learning of Humanoid Robot Behavior from Natural Interaction and Large Language Models

Learning Deep Features for Robotic Inference from Physical Interactions.

Spatially Visual Perception for End-to-End Robotic Learning

Incremental Learning Techniques for Online Human Activity Recognition

A Robotic Visual Grasping Design: Rethinking Convolution Neural Network with High-Resolutions

Improving Visual Perception of a Social Robot for Controlled and In-the-wild Human-robot Interaction

A neural network based on biological vision learning and its application on robot

An Incremental Learning-Based Mechanism for Object Recognition in Cloud Robotic System

Incremental Learning for Robot Shared Autonomy