Abstract:When human acquire physical skills (e.g., tennis) from experts, we tend to first learn from merely observing the expert. But this is often insufficient. We then engage in practice, where we try to emulate the expert and ensure that our actions produce similar effects on our environment. Inspired by this observation, we introduce Combining IMitation and Emulation for Motion Refinement (CIMER) -- a two-stage framework to learn dexterous prehensile manipulation skills from state-only observations. CIMER's first stage involves imitation: simultaneously encode the complex interdependent motions of the robot hand and the object in a structured dynamical system. This results in a reactive motion generation policy that provides a reasonable motion prior, but lacks the ability to reason about contact effects due to the lack of action labels. The second stage involves emulation: learn a motion refinement policy via reinforcement that adjusts the robot hand's motion prior such that the desired object motion is reenacted. CIMER is both task-agnostic (no task-specific reward design or shaping) and intervention-free (no additional teleoperated or labeled demonstrations). Detailed experiments with prehensile dexterity reveal that i) imitation alone is insufficient, but adding emulation drastically improves performance, ii) CIMER outperforms existing methods in terms of sample efficiency and the ability to generate realistic and stable motions, iii) CIMER can either zero-shot generalize or learn to adapt to novel objects from the YCB dataset, even outperforming expert policies trained with action labels in most cases. Source code and videos are available at <a class="link-external link-https" href="https://sites.google.com/view/cimer-2024/" rel="external noopener nofollow">this https URL</a>.

BiKC: Keypose-Conditioned Consistency Policy for Bimanual Robotic Manipulation

Robust and High-Precision End-to-End Control Policy for Multi-stage Manipulation Task with Behavioral Cloning.

Bi-KVIL: Keypoints-based Visual Imitation Learning of Bimanual Manipulation Tasks

Dexterous Manipulation Control of a Bionic Prosthesis in Cooperation with Human Upper Limbs

A System for Imitation Learning of Contact-Rich Bimanual Manipulation Policies

Stabilize to Act: Learning to Coordinate for Bimanual Manipulation

Learning Diverse Bimanual Dexterous Manipulation Skills from Human Demonstrations

One-Shot Imitation Learning with Invariance Matching for Robotic Manipulation

A Learning-based Adaptive Compliance Method for Symmetric Bi-manual Manipulation

Object-Centric Dexterous Manipulation from Human Motion Data

Concept2Robot: Learning Manipulation Concepts from Instructions and Human Demonstrations

InterACT: Inter-dependency Aware Action Chunking with Hierarchical Attention Transformers for Bimanual Manipulation

Efficient Bimanual Handover and Rearrangement Via Symmetry-Aware Actor-Critic Learning.

Learning Prehensile Dexterity by Imitating and Emulating State-only Observations

Learning Manipulation by Predicting Interaction

DAIR: Disentangled Attention Intrinsic Regularization for Safe and Efficient Bimanual Manipulation

On the Utility of Koopman Operator Theory in Learning Dexterous Manipulation Skills

From One Hand to Multiple Hands: Imitation Learning for Dexterous Manipulation From Single-Camera Teleoperation

Waypoint-Based Imitation Learning for Robotic Manipulation

Learning Dexterous Manipulation Policies from Experience and Imitation

K-VIL: Keypoints-based Visual Imitation Learning