Abstract:Highlights • A parametric shape descriptor to efficiently characterize 3D deformations based on online curve/surface fitting. • A robust shape prediction network based on adversarial neural networks to compensate visual occlusions. • An optimization-based estimator to approximate the deformation Jacobian matrix and satisfy various performance constraints. • An MPC-based controller to guide the shaping motions while simultaneously solving saturation, workspace, and obstacle constraints. Background: The manipulation of compliant objects by robotic systems remains a challenging task, largely due to their variable shapes and the complex, high-dimensional nature of their interaction dynamics. Traditional robotic manipulation strategies struggle with the accurate modeling and control necessary to handle such materials, especially in the presence of visual occlusions that frequently occur in dynamic environments. Meanwhile, for most unstructured environments, robots are required to have autonomous interactions with their surroundings. Methods: To solve the shape manipulation of compliant objects in an unstructured environment, we begin by exploring the regression-based algorithm of representing the high-dimensional configuration space of deformable objects in a compressed form that enables efficient and effective manipulation. Simultaneously, we address the issue of visual occlusions by proposing the integration of an adversarial network, enabling guiding the shaping task even with partial observations of the object. Afterwards, we propose a receding-time estimator to coordinate the robot action with the computed shape features while satisfying various performance criteria. Finally, model predictive controller is utilized to compute the robot's shaping motions subject to safety constraints. Detailed experiments are presented to evaluate the proposed manipulation framework. Significant findings: Our MPC framework utilizes the compressed representation and occlusion-compensated information to predict the object's behavior, while the multi-objective optimizer ensures that the resulting control actions meet multiple performance criteria. Through rigorous experimental validation, our approach demonstrates superior manipulation capabilities in scenarios with visual obstructions, outperforming existing methods in terms of precision and operational reliability. The findings highlight the potential of our integrated approach to significantly enhance the manipulation of compliant objects in real-world robotic applications.

Dense-Descriptor Transporter Networks for Multidimensional Deformable Object Manipulation

Multidimensional Deformable Object Manipulation Based on DN-Transporter Networks

Multi-Dimensional Deformable Object Manipulation Using Equivariant Models.

Learning Sim-to-Real Dense Object Descriptors for Robotic Manipulation

Learning to Rearrange Deformable Cables, Fabrics, and Bags with Goal-Conditioned Transporter Networks

Learning Dense Visual Object Descriptors to Fold Two-Dimensional Deformable Fabrics

Graph-Transporter: A Graph-based Learning Method for Goal-Conditioned Deformable Object Rearranging Task

Learning Latent Dynamics for Autonomous Shape Control of Deformable Object

DeformerNet: A Deep Learning Approach to 3D Deformable Object Manipulation

Learning to Estimate 3-D States of Deformable Linear Objects from Single-Frame Occluded Point Clouds

DeformNet: Latent Space Modeling and Dynamics Prediction for Deformable Object Manipulation

DeformerNet: Learning Bimanual Manipulation of 3D Deformable Objects

Learning Correspondence for Deformable Objects

Category-Level 6-D Object Pose Estimation With Shape Deformation for Robotic Grasp Detection

Learning Language-Conditioned Deformable Object Manipulation with Graph Dynamics

Bimanual Deformable Bag Manipulation Using a Structure-of-Interest Based Neural Dynamics Model

MoDA: Modeling Deformable 3D Objects from Casual Videos

Model predictive manipulation of compliant objects with multi-objective optimizer and adversarial network for occlusion compensation

DTF-Net: Category-Level Pose Estimation and Shape Reconstruction via Deformable Template Field

3D_DEN: Open-ended 3D Object Recognition using Dynamically Expandable Networks

Self-supervised Learning of Implicit Shape Representation with Dense Correspondence for Deformable Objects