Abstract:Humans are able to seamlessly visually imitate others, by inferring their intentions and using past experience to achieve the same end goal. In other words, we can parse complex semantic knowledge from raw video and efficiently translate that into concrete motor control. Is it possible to give a robot this same capability? Prior research in robot imitation learning has created agents which can acquire diverse skills from expert human operators. However, expanding these techniques to work with a single positive example during test time is still an open challenge. Apart from control, the difficulty stems from mismatches between the demonstrator and robot domains. For example, objects may be placed in different locations (e.g. kitchen layouts are different in every house). Additionally, the demonstration may come from an agent with different morphology and physical appearance (e.g. human), so one-to-one action correspondences are not available. This paper investigates techniques which allow robots to partially bridge these domain gaps, using their past experience. A neural network is trained to mimic ground truth robot actions given context video from another agent, and must generalize to unseen task instances when prompted with new videos during test time. We hypothesize that our policy representations must be both context driven and dynamics aware in order to perform these tasks. These assumptions are baked into the neural network using the Transformers attention mechanism and a self-supervised inverse dynamics loss. Finally, we experimentally determine that our method accomplishes a $\sim 2$x improvement in terms of task success rate over prior baselines in a suite of one-shot manipulation tasks.

Explicit-to-Implicit Robot Imitation Learning by Exploring Visual Content Change

Learning Robot Manipulation Skills from Human Demonstration Videos Using Two-Stream 2-D/3-D Residual Networks with Self-Attention

Visual Imitation Made Easy

InfoGAIL: Interpretable Imitation Learning from Visual Demonstrations

Robotic Imitation of Human Actions

Manipulator-Independent Representations for Visual Imitation

Keyframe-Focused Visual Imitation Learning

Extraneousness-Aware Imitation Learning

MimicPlay: Long-Horizon Imitation Learning by Watching Human Play

K-VIL: Keypoints-based Visual Imitation Learning

Giving Robots a Hand: Learning Generalizable Manipulation with Eye-in-Hand Human Video Demonstrations

Towards Learning to Imitate from a Single Video Demonstration

Zero-shot Imitation Learning from Demonstrations for Legged Robot Visual Navigation

Learning Actions from Human Demonstration Video for Robotic Manipulation

VLMimic: Vision Language Models are Visual Imitation Learner for Fine-grained Actions

Transformers for One-Shot Visual Imitation

Manipulate by Seeing: Creating Manipulation Controllers from Pre-Trained Representations

An Empirical Investigation of Representation Learning for Imitation

Imitator Learning: Achieve Out-of-the-Box Imitation Ability in Variable Environments

Robust Visual Imitation Learning with Inverse Dynamics Representations

Vision-based Robotic Arm Imitation by Human Gesture