Abstract:Deformable object manipulation stands as one of the most captivating yet formidable challenges in robotics. While previous techniques have predominantly relied on learning latent dynamics through demonstrations, typically represented as either particles or images, there exists a pertinent limitation: acquiring suitable demonstrations, especially for long-horizon tasks, can be elusive. Moreover, basing learning entirely on demonstrations can hamper the model's ability to generalize beyond the demonstrated tasks. In this work, we introduce a demonstration-free hierarchical planning approach capable of tackling intricate long-horizon tasks without necessitating any training. We employ large language models (LLMs) to articulate a high-level, stage-by-stage plan corresponding to a specified task. For every individual stage, the LLM provides both the tool's name and the Python code to craft intermediate subgoal point clouds. With the tool and subgoal for a particular stage at our disposal, we present a granular closed-loop model predictive control strategy. This leverages Differentiable Physics with Point-to-Point correspondence (DiffPhysics-P2P) loss in the earth mover distance (EMD) space, applied iteratively. Experimental findings affirm that our technique surpasses multiple benchmarks in dough manipulation, spanning both short and long horizons. Remarkably, our model demonstrates robust generalization capabilities to novel and previously unencountered complex tasks without any preliminary demonstrations. We further substantiate our approach with experimental trials on real-world robotic platforms. Our project page: <a class="link-external link-https" href="https://qq456cvb.github.io/projects/donut" rel="external noopener nofollow">this https URL</a>.

A Human Demonstration Based Inverse Dynamics Model for Long-Sequential Dough Shape Manipulation

Human Demonstration Trajectory Refinement for Redundant Manipulators.

Learning Goal-oriented Bimanual Dough Rolling Using Dynamic Heterogeneous Graph Based on Human Demonstration

COMBINATION OF AFFINE DEFORMATION AND DYNAMIC MOVEMENT PRIMITIVE IN LEARNING HUMAN MOTION FOR REDUNDANT MANIPULATOR

Make a Donut: Language-Guided Hierarchical EMD-Space Planning for Zero-shot Deformable Object Manipulation

Make a Donut: Hierarchical EMD-Space Planning for Zero-Shot Deformable Manipulation with Tools

Learning Closed-loop Dough Manipulation Using a Differentiable Reset Module

Robotic Dough Shaping

Plasti-Former: an Optimal Transport Enhanced Transformer Based Inverse Dynamics Model for Plasticine Shape Manipulation

Control-oriented modeling of flight demonstrations for quadrotors using higher-order statistics and dynamic movement primitives

DoughNet: A Visual Predictive Model for Topological Manipulation of Deformable Objects

Diff-LfD: Contact-aware Model-based Learning from Visual Demonstration for Robotic Manipulation Via Differentiable Physics-based Simulation and Rendering.

Learning from Demonstration for 7-DOF Anthropomorphic Manipulators Without Offset Via Analytical Inverse Kinematics

Learning From Human Demonstrations for Wheel Mobile Manipulator: An Unscented Model Predictive Control Approach

Learning Cooperative Dynamic Manipulation Skills from Human Demonstration Videos

Learning Forceful Manipulation Skills from Multi-modal Human Demonstrations

Design of Demonstration-Driven Assembling Manipulator

Learning to Manipulate Amorphous Materials

Dynamics-Guided Diffusion Model for Robot Manipulator Design

Learning Latent Actions to Control Assistive Robots

Deep Dynamics Models for Learning Dexterous Manipulation