Abstract:Unlike quasi-static robotic manipulation tasks like pick-and-place, dynamic tasks such as non-prehensile manipulation pose greater challenges, especially for vision-based control. Successful control requires the extraction of features relevant to the target task. In visual imitation learning settings, these features can be learnt by backpropagating the policy loss through the vision backbone. Yet, this approach tends to learn task-specific features with limited generalizability. Alternatively, learning world models can realize more generalizable vision backbones. Utilizing the learnt features, task-specific policies are subsequently trained. Commonly, these models are trained solely to predict the next RGB state from the current state and action taken. But only-RGB prediction might not fully-capture the task-relevant dynamics. In this work, we hypothesize that direct supervision of target dynamic states (Dynamics Mapping) can learn better dynamics-informed world models. Beside the next RGB reconstruction, the world model is also trained to directly predict position, velocity, and acceleration of environment rigid bodies. To verify our hypothesis, we designed a non-prehensile 2D environment tailored to two tasks: "Balance-Reaching" and "Bin-Dropping". When trained on the first task, dynamics mapping enhanced the task performance under different training configurations (Decoupled, Joint, End-to-End) and policy architectures (Feedforward, Recurrent). Notably, its most significant impact was for world model pretraining boosting the success rate from 21% to 85%. Although frozen dynamics-informed world models could generalize well to a task with in-domain dynamics, but poorly to a one with out-of-domain dynamics.

Learning Predictive Representations for Deformable Objects Using Contrastive Estimation

Learning to Manipulate Deformable Objects without Demonstrations

Learning Latent Dynamics for Autonomous Shape Control of Deformable Object

Learning Foresightful Dense Visual Affordance for Deformable Object Manipulation

DeformNet: Latent Space Modeling and Dynamics Prediction for Deformable Object Manipulation

DiffSRL: Learning Dynamical State Representation for Deformable Object Manipulation With Differentiable Simulation

Learning Correspondence for Deformable Objects

Learning Robotic Manipulation through Visual Planning and Acting

Learning deformable linear object dynamics from a single trajectory

Learning Language-Conditioned Deformable Object Manipulation with Graph Dynamics

Dynamic Cloth Folding Using Curriculum Learning

Global Model Learning for Large Deformation Control of Elastic Deformable Linear Objects: An Efficient and Adaptive Approach

DiffSRL: Learning Dynamical State Representation for Deformable Object Manipulation with Differentiable Simulator

Iterative Residual Policy: for Goal-Conditioned Dynamic Manipulation of Deformable Objects

Visual Imitation Learning of Non-Prehensile Manipulation Tasks with Dynamics-Supervised Models

Manipulate by Seeing: Creating Manipulation Controllers from Pre-Trained Representations

DeformPAM: Data-Efficient Learning for Long-horizon Deformable Object Manipulation via Preference-based Action Alignment

Differentiable Discrete Elastic Rods for Real-Time Modeling of Deformable Linear Objects

Model-Free Visually Servoed Deformation Control of Elastic Objects by Robot Manipulators

Learning-based Feedback Controller for Deformable Object Manipulation

Multi-Dimensional Deformable Object Manipulation Using Equivariant Models.