Abstract:Extrinsic manipulation, the use of environment contacts to achieve manipulation objectives, enables strategies that are otherwise impossible with a parallel jaw gripper. However, orchestrating a long-horizon sequence of contact interactions between the robot, object, and environment is notoriously challenging due to the scene diversity, large action space, and difficult contact dynamics. We observe that most extrinsic manipulation are combinations of short-horizon primitives, each of which depend strongly on initializing from a desirable contact configuration to succeed. Therefore, we propose to generalize one extrinsic manipulation trajectory to diverse objects and environments by retargeting contact requirements. We prepare a single library of robust short-horizon, goal-conditioned primitive policies, and design a framework to compose state constraints stemming from contacts specifications of each primitive. Given a test scene and a single demo prescribing the primitive sequence, our method enforces the state constraints on the test scene and find intermediate goal states using inverse kinematics. The goals are then tracked by the primitive policies. Using a 7+1 DoF robotic arm-gripper system, we achieved an overall success rate of 80.5% on hardware over 4 long-horizon extrinsic manipulation tasks, each with up to 4 primitives. Our experiments cover 10 objects and 6 environment configurations. We further show empirically that our method admits a wide range of demonstrations, and that contact retargeting is indeed the key to successfully combining primitives for long-horizon extrinsic manipulation. Code and additional details are available at

One-Shot Manipulation Strategy Learning by Making Contact Analogies

Learning Reusable Manipulation Strategies

One-Shot Transfer of Long-Horizon Extrinsic Manipulation Through Contact Retargeting

Ag2Manip: Learning Novel Manipulation Skills with Agent-Agnostic Visual and Action Representations

One-Shot Imitation Learning with Invariance Matching for Robotic Manipulation

One-Shot Imitation from Observing Humans via Domain-Adaptive Meta-Learning

Learning Manipulation by Predicting Interaction

Pixels to Plans: Learning Non-Prehensile Manipulation by Imitating a Planner

Towards Generalizable Zero-Shot Manipulation via Translating Human Interaction Plans

Learning to Transfer In-Hand Manipulations Using a Greedy Shape Curriculum

Manipulation Pattern Discovery: A Nonparametric Bayesian Approach

Manipulate by Seeing: Creating Manipulation Controllers from Pre-Trained Representations

Where2Explore: Few-shot Affordance Learning for Unseen Novel Categories of Articulated Objects

Object-Centric Dexterous Manipulation from Human Motion Data

Efficient Robot Skill Learning with Imitation from a Single Video for Contact-Rich Fabric Manipulation

Learning Generalizable Dexterous Manipulation from Human Grasp Affordance

Learning to Manipulate Tools by Aligning Simulation to Video Demonstration

Learning to Design and Use Tools for Robotic Manipulation

Learning to Scaffold the Development of Robotic Manipulation Skills

Zero-Shot Transfer of Haptics-Based Object Insertion Policies

Learning Extrinsic Dexterity with Parameterized Manipulation Primitives