Abstract:Humans are able to seamlessly visually imitate others, by inferring their intentions and using past experience to achieve the same end goal. In other words, we can parse complex semantic knowledge from raw video and efficiently translate that into concrete motor control. Is it possible to give a robot this same capability? Prior research in robot imitation learning has created agents which can acquire diverse skills from expert human operators. However, expanding these techniques to work with a single positive example during test time is still an open challenge. Apart from control, the difficulty stems from mismatches between the demonstrator and robot domains. For example, objects may be placed in different locations (e.g. kitchen layouts are different in every house). Additionally, the demonstration may come from an agent with different morphology and physical appearance (e.g. human), so one-to-one action correspondences are not available. This paper investigates techniques which allow robots to partially bridge these domain gaps, using their past experience. A neural network is trained to mimic ground truth robot actions given context video from another agent, and must generalize to unseen task instances when prompted with new videos during test time. We hypothesize that our policy representations must be both context driven and dynamics aware in order to perform these tasks. These assumptions are baked into the neural network using the Transformers attention mechanism and a self-supervised inverse dynamics loss. Finally, we experimentally determine that our method accomplishes a $\sim 2$x improvement in terms of task success rate over prior baselines in a suite of one-shot manipulation tasks.

One-Shot Imitation Drone Filming of Human Motion Videos

Imitation Learning-Based Algorithm for Drone Cinematography System

Learning to Capture a Film-Look Video with a Camera Drone

Path-Analysis-Based Reinforcement Learning Algorithm for Imitation Filming

Flow-Guided Single Object Tracking Framework in UAV Aerial Video

Learning to Film from Professional Human Motion Videos

One-Shot Imitation from Observing Humans via Domain-Adaptive Meta-Learning

Transformers for One-Shot Visual Imitation

One-Shot Domain-Adaptive Imitation Learning via Progressive Learning

Imitation Learning of Robotic Arm with Hierarchical Training Based on Human Videos

Can We Enable the Drone to be a Filmmaker?

Learn by Observation: Imitation Learning for Drone Patrolling from Videos of A Human Navigator.

Act: An Autonomous Drone Cinematography System For Action Scenes

Learning One-Shot Imitation From Humans Without Humans

Through-the-Lens Drone Filming.

One-Shot Imitation Learning: A Pose Estimation Perspective

Auto Filmer: Autonomous Aerial Videography Under Human Interaction

Object Detection-Based One-Shot Imitation Learning with an RGB-D Camera

One-shot imitation learning

Restore DeepFakes Video Frames Via Identifying Individual Motion Styles

CineTransfer: Controlling a Robot to Imitate Cinematographic Style from a Single Example