Abstract:We present a data-driven method for deformation capture and modeling of general soft objects. We adopt an iterative framework that consists of one component for physics-based deformation tracking and another for spacetime optimization of deformation parameters. Low cost depth sensors are used for the deformation capture, and we do not require any force-displacement measurements, thus making the data capture a cheap and convenient process. We augment a state-of-the-art probabilistic tracking method to robustly handle noise, occlusions, fast movements and large deformations. The spacetime optimization aims to match the simulated trajectories with the tracked ones. The optimized deformation model is then used to boost the accuracy of the tracking results, which can in turn improve the deformation parameter estimation itself in later iterations. Numerical experiments demonstrate that the tracking and parameter optimization components complement each other nicely. Our spacetime optimization of the deformation model includes not only the material elasticity parameters and dynamic damping coefficients, but also the reference shape which can differ significantly from the static shape for soft objects. The resulting optimization problem is highly nonlinear in high dimensions, and challenging to solve with previous methods. We propose a novel splitting algorithm that alternates between reference shape optimization and deformation parameter estimation, and thus enables tailoring the optimization of each subproblem more efficiently and robustly. Our system enables realistic motion reconstruction as well as synthesis of virtual soft objects in response to user stimulation. Validation experiments show that our method not only is accurate, but also compares favorably to existing techniques. We also showcase the ability of our system with high quality animations generated from optimized deformation parameters for a variety of soft objects, such as live plants and fabricated models.

DeformNet: Latent Space Modeling and Dynamics Prediction for Deformable Object Manipulation

DeformerNet: Learning Bimanual Manipulation of 3D Deformable Objects

DeformerNet: A Deep Learning Approach to 3D Deformable Object Manipulation

Learning Predictive Representations for Deformable Objects Using Contrastive Estimation

MoDA: Modeling Deformable 3D Objects from Casual Videos

DEFO-NET: Learning Body Deformation Using Generative Adversarial Networks.

DeformPAM: Data-Efficient Learning for Long-horizon Deformable Object Manipulation via Preference-based Action Alignment

DiffSRL: Learning Dynamical State Representation for Deformable Object Manipulation With Differentiable Simulation

DoughNet: A Visual Predictive Model for Topological Manipulation of Deformable Objects

Learning Latent Dynamics for Autonomous Shape Control of Deformable Object

Multi-Dimensional Deformable Object Manipulation Using Equivariant Models.

Learning Foresightful Dense Visual Affordance for Deformable Object Manipulation

Differentiable Discrete Elastic Rods for Real-Time Modeling of Deformable Linear Objects

Deformation capture and modeling of soft objects

Reactive human–robot collaborative manipulation of deformable linear objects using a new topological latent control model

DexDeform: Dexterous Deformable Object Manipulation with Human Demonstrations and Differentiable Physics

Bimanual Deformable Bag Manipulation Using a Structure-of-Interest Based Neural Dynamics Model

Global Model Learning for Large Deformation Control of Elastic Deformable Linear Objects: An Efficient and Adaptive Approach

Multidimensional Deformable Object Manipulation Based on DN-Transporter Networks

Shape-Space Deformer: Unified Visuo-Tactile Representations for Robotic Manipulation of Deformable Objects

Neural Material: Learning Elastic Constitutive Material and Damping Models from Sparse Data