Abstract:We present a data-driven method for deformation capture and modeling of general soft objects. We adopt an iterative framework that consists of one component for physics-based deformation tracking and another for spacetime optimization of deformation parameters. Low cost depth sensors are used for the deformation capture, and we do not require any force-displacement measurements, thus making the data capture a cheap and convenient process. We augment a state-of-the-art probabilistic tracking method to robustly handle noise, occlusions, fast movements and large deformations. The spacetime optimization aims to match the simulated trajectories with the tracked ones. The optimized deformation model is then used to boost the accuracy of the tracking results, which can in turn improve the deformation parameter estimation itself in later iterations. Numerical experiments demonstrate that the tracking and parameter optimization components complement each other nicely. Our spacetime optimization of the deformation model includes not only the material elasticity parameters and dynamic damping coefficients, but also the reference shape which can differ significantly from the static shape for soft objects. The resulting optimization problem is highly nonlinear in high dimensions, and challenging to solve with previous methods. We propose a novel splitting algorithm that alternates between reference shape optimization and deformation parameter estimation, and thus enables tailoring the optimization of each subproblem more efficiently and robustly. Our system enables realistic motion reconstruction as well as synthesis of virtual soft objects in response to user stimulation. Validation experiments show that our method not only is accurate, but also compares favorably to existing techniques. We also showcase the ability of our system with high quality animations generated from optimized deformation parameters for a variety of soft objects, such as live plants and fabricated models.

Fast Model-Based Contact Patch and Pose Estimation for Highly Deformable Dense-Geometry Tactile Sensors

Proximity and Visuotactile Point Cloud Fusion for Contact Patches in Extreme Deformation

A Novel Inverse Solution Of Contact Force Based On A Sparse Tactile Sensor Array

Soft Contact Simulation and Manipulation Learning of Deformable Objects with Vision-based Tactile Sensor

Manipulation via Membranes: High-Resolution and Highly Deformable Tactile Sensing and Control

Touch and deformation perception of soft manipulators with capacitive e-skins and deep learning

Soft-bubble: A highly compliant dense geometry tactile sensor for robot manipulation

9DTact: A Compact Vision-Based Tactile Sensor for Accurate 3D Shape Reconstruction and Generalizable 6D Force Estimation

RoboPack: Learning Tactile-Informed Dynamics Models for Dense Packing

Robust Affordable 3D Haptic Sensation via Learning Deformation Patterns

Robust Learning of Tactile Force Estimation through Robot Interaction

DenseTact 2.0: Optical Tactile Sensor for Shape and Force Reconstruction

Model-Based Disturbance Estimation for a Fiber-Reinforced Soft Manipulator using Orientation Sensing

Real-Time Deformable-Contact-Aware Model Predictive Control for Force-Modulated Manipulation

Realtime State Estimation with Tactile and Visual Sensing for Inserting a Suction-held Object

Model-Based 3D Contact Geometry Perception for Visual Tactile Sensor

ConTac: Continuum-Emulated Soft Skinned Arm with Vision-based Shape Sensing and Contact-aware Manipulation

Deformation capture and modeling of soft objects

FeelAnyForce: Estimating Contact Force Feedback from Tactile Sensation for Vision-Based Tactile Sensors

A Distributed Force Reconstruction Mapping Model with Physical Compactness and Low Computational Cost for Vision-based Tactile Sensors

Tactile Estimation of Extrinsic Contact Patch for Stable Placement