Abstract:Visual imitation learning is a promising approach that promotes robots to learn skills from visual demonstrations. However, current visual imitation learning approaches introduce unreasonable assumptions that the contexts of the visual demonstrations and the robot observations are consistent, which affects the flexibility and scalability of the approaches. It is a key challenge for robots to learn from visual demonstrations with inconsistent contexts. Inconsistent contexts may cause a serious difference in the pixel distribution of the operator and the environment, which makes vision-based control policies hardly effective. In this paper, we propose a novel imitation learning framework to enable robots to reproduce behavior by watching human demonstrations with inconsistent contexts, such as different viewpoints, operators, backgrounds, object appearances and positions. Specifically, our framework consists of three networks: flow-based viewpoint transformation network (FVTrans), robot2human alignment network (RANet) and inverse dynamics network (IDNet). First, FVTrans transforms various third-person demonstrations into the fixed robot execution view. With a meta learning strategy, FVTrans can quickly adapt to novel contexts with few samples. Then, RANet aligns the human and the robot at the feature level. Therefore, the demonstration feature can be used as a subgoal of the current moment. Finally, IDNet predicts the joint angles of the robot. We collect a multi-context dataset on the real robot (UR5) for three tasks, including grasping cups, sweeping garbage and placing objects. We empirically demonstrate that our framework can perform three tasks with a high success rate and be effectively generalized to different contexts.

Simultaneous Multi-task Determination And Manipulation With Vision-based Self-reasoning Network

Learning Robot Manipulation Skills from Human Demonstration Videos Using Two-Stream 2-D/3-D Residual Networks with Self-Attention

A Single Multi-Task Deep Neural Network with a Multi-Scale Feature Aggregation Mechanism for Manipulation Relationship Reasoning in Robotic Grasping

Vision-Based Multi-Task Manipulation for Inexpensive Robots Using End-To-End Learning from Demonstration

Leveraging the Efficiency of Multi-Task Robot Manipulation Via Task-Evoked Planner and Reinforcement Learning

RPRG: Toward Real-time Robotic Perception, Reasoning and Grasping with One Multi-task Convolutional Neural Network.

Enhancing Robot Manipulation Skill Learning with Multi-task Capability Based on Transformer and Token Reduction

Vision-Based Categorical Object Pose Estimation and Manipulation.

A Semantic Robotic Grasping Framework Based on Multi-Task Learning in Stacking Scenes.

End-to-end Multi-Instance Robotic Reaching from Monocular Vision

Visual Manipulation Relationship Network for Autonomous Robotics

PRRM: An Efficient Framework for Learning Multi-step Robotic Manipulation Tasks

Robot Learning from Human Demonstrations with Inconsistent Contexts

Composable Instructions and Prospection Guided Visuomotor Control for Robotic Manipulation.

A Knowledge-Based Task Planning Approach for Robot Multi-Task Manipulation

Mixline: A Hybrid Reinforcement Learning Framework for Long-horizon Bimanual Coffee Stirring Task

A Multi-task Convolutional Neural Network for Autonomous Robotic Grasping in Object Stacking Scenes

Play to the Score: Stage-Guided Dynamic Multi-Sensory Fusion for Robotic Manipulation

Multifunctional Robot Based on Multimodal Brain-Machine Interface