Abstract:Visual imitation learning is a promising approach that promotes robots to learn skills from visual demonstrations. However, current visual imitation learning approaches introduce unreasonable assumptions that the contexts of the visual demonstrations and the robot observations are consistent, which affects the flexibility and scalability of the approaches. It is a key challenge for robots to learn from visual demonstrations with inconsistent contexts. Inconsistent contexts may cause a serious difference in the pixel distribution of the operator and the environment, which makes vision-based control policies hardly effective. In this paper, we propose a novel imitation learning framework to enable robots to reproduce behavior by watching human demonstrations with inconsistent contexts, such as different viewpoints, operators, backgrounds, object appearances and positions. Specifically, our framework consists of three networks: flow-based viewpoint transformation network (FVTrans), robot2human alignment network (RANet) and inverse dynamics network (IDNet). First, FVTrans transforms various third-person demonstrations into the fixed robot execution view. With a meta learning strategy, FVTrans can quickly adapt to novel contexts with few samples. Then, RANet aligns the human and the robot at the feature level. Therefore, the demonstration feature can be used as a subgoal of the current moment. Finally, IDNet predicts the joint angles of the robot. We collect a multi-context dataset on the real robot (UR5) for three tasks, including grasping cups, sweeping garbage and placing objects. We empirically demonstrate that our framework can perform three tasks with a high success rate and be effectively generalized to different contexts.

Robot Learning With A Spatial, Temporal, And Causal And-Or Graph

Learning Robot Manipulation Skills from Human Demonstration Videos Using Two-Stream 2-D/3-D Residual Networks with Self-Attention

GSC: A Graph-Based Skill Composition Framework for Robot Learning

Towards Learning from Demonstration System for Parts Assembly: A Graph Based Representation for Knowledge

Graph-based Hierarchical Knowledge Representation for Robot Task Transfer from Virtual to Physical World

Robot Planning Based on Behavior Tree and Knowledge Graph

Jointly Learning Grounded Task Structures from Language Instruction and Visual Demonstration.

Task Learning Through Visual Demonstration and Situated Dialogue.

Learning Robotic Manipulation through Visual Planning and Acting

Robot Learning from Human Demonstrations with Inconsistent Contexts

A Task-Learning Strategy for Robotic Assembly Tasks from Human Demonstrations

Modeling Long-horizon Tasks as Sequential Interaction Landscapes

Long-term Robot Manipulation Task Planning with Scene Graph and Semantic Knowledge

Hierarchical Understanding in Robotic Manipulation: A Knowledge-Based Framework

Learning Responsive Humanoid Motion Skills from Graph-Powered Motion Matching

Exploring CausalWorld: Enhancing robotic manipulation via knowledge transfer and curriculum learning

Extracting and Transferring Hierarchical Knowledge to Robots Using Virtual Reality.

Human Demonstrations are Generalizable Knowledge for Robots

Learning Crowd Behaviors in Navigation with Attention-based Spatial-Temporal Graphs

$π_0$: A Vision-Language-Action Flow Model for General Robot Control

Learning Cloth Folding Tasks with Refined Flow Based Spatio-Temporal Graphs