Abstract:Visual imitation learning is a promising approach that promotes robots to learn skills from visual demonstrations. However, current visual imitation learning approaches introduce unreasonable assumptions that the contexts of the visual demonstrations and the robot observations are consistent, which affects the flexibility and scalability of the approaches. It is a key challenge for robots to learn from visual demonstrations with inconsistent contexts. Inconsistent contexts may cause a serious difference in the pixel distribution of the operator and the environment, which makes vision-based control policies hardly effective. In this paper, we propose a novel imitation learning framework to enable robots to reproduce behavior by watching human demonstrations with inconsistent contexts, such as different viewpoints, operators, backgrounds, object appearances and positions. Specifically, our framework consists of three networks: flow-based viewpoint transformation network (FVTrans), robot2human alignment network (RANet) and inverse dynamics network (IDNet). First, FVTrans transforms various third-person demonstrations into the fixed robot execution view. With a meta learning strategy, FVTrans can quickly adapt to novel contexts with few samples. Then, RANet aligns the human and the robot at the feature level. Therefore, the demonstration feature can be used as a subgoal of the current moment. Finally, IDNet predicts the joint angles of the robot. We collect a multi-context dataset on the real robot (UR5) for three tasks, including grasping cups, sweeping garbage and placing objects. We empirically demonstrate that our framework can perform three tasks with a high success rate and be effectively generalized to different contexts.

Robot Imitation Learning from Image-Only Observation Without Real-World Interaction

HILONet: Hierarchical Imitation Learning from Non-Aligned Observations

Robot imitation from multimodal observation with unsupervised cross-modal representation

Robot Learning from Human Demonstrations with Inconsistent Contexts

Imitation Learning from Observation with Automatic Discount Scheduling

Visual IRL for Human-Like Robotic Manipulation

Interactive Imitation Learning in Robotics based on Simulations

Generalized Robot Learning Framework

Learning from demonstrations: An intuitive VR environment for imitation learning of construction robots

On the Guaranteed Almost Equivalence Between Imitation Learning from Observation and Demonstration

Practical Imitation Learning in the Real World via Task Consistency Loss

Robot Skill Learning Based on Interacting with RGB-D Image

Demonstration Guided Actor-Critic Deep Reinforcement Learning for Fast Teaching of Robots in Dynamic Environments

Learning One-Shot Imitation From Humans Without Humans

Curriculum-based Sensing Reduction in Simulation to Real-World Transfer for In-hand Manipulation

Imitator Learning: Achieve Out-of-the-Box Imitation Ability in Variable Environments

Learning Latent Object-Centric Representations for Visual-Based Robot Manipulation

Interactive Robot Learning from Verbal Correction

A novel simulation reality closed loop learning framework for autonomous robot skill learning

Incremental learning of high-level concepts by imitation

Training of construction robots using imitation learning and environmental rewards