Abstract:Highlights • A parametric shape descriptor to efficiently characterize 3D deformations based on online curve/surface fitting. • A robust shape prediction network based on adversarial neural networks to compensate visual occlusions. • An optimization-based estimator to approximate the deformation Jacobian matrix and satisfy various performance constraints. • An MPC-based controller to guide the shaping motions while simultaneously solving saturation, workspace, and obstacle constraints. Background: The manipulation of compliant objects by robotic systems remains a challenging task, largely due to their variable shapes and the complex, high-dimensional nature of their interaction dynamics. Traditional robotic manipulation strategies struggle with the accurate modeling and control necessary to handle such materials, especially in the presence of visual occlusions that frequently occur in dynamic environments. Meanwhile, for most unstructured environments, robots are required to have autonomous interactions with their surroundings. Methods: To solve the shape manipulation of compliant objects in an unstructured environment, we begin by exploring the regression-based algorithm of representing the high-dimensional configuration space of deformable objects in a compressed form that enables efficient and effective manipulation. Simultaneously, we address the issue of visual occlusions by proposing the integration of an adversarial network, enabling guiding the shaping task even with partial observations of the object. Afterwards, we propose a receding-time estimator to coordinate the robot action with the computed shape features while satisfying various performance criteria. Finally, model predictive controller is utilized to compute the robot's shaping motions subject to safety constraints. Detailed experiments are presented to evaluate the proposed manipulation framework. Significant findings: Our MPC framework utilizes the compressed representation and occlusion-compensated information to predict the object's behavior, while the multi-objective optimizer ensures that the resulting control actions meet multiple performance criteria. Through rigorous experimental validation, our approach demonstrates superior manipulation capabilities in scenarios with visual obstructions, outperforming existing methods in terms of precision and operational reliability. The findings highlight the potential of our integrated approach to significantly enhance the manipulation of compliant objects in real-world robotic applications.

3D Deformable Object Manipulation using Fast Online Gaussian Process Regression

Learning-based Feedback Controller for Deformable Object Manipulation

Global Model Learning for Large Deformation Control of Elastic Deformable Linear Objects: An Efficient and Adaptive Approach

Adaptive Shape Servoing of Elastic Rods using Parameterized Regression Features and Auto-Tuning Motion Controls

A Motion Planning Method for Visual Servoing Using Deep Reinforcement Learning in Autonomous Robotic Assembly

Adaptive Online Learning and Robust 3-D Shape Servoing of Continuum and Soft Robots in Unstructured Environments

DeformGS: Scene Flow in Highly Deformable Scenes for Deformable Object Manipulation

Hand-eye Robotic Visual Servoing Around Moving Objects Using Active Deformable Models

Adaptive Control for Robotic Manipulation of Deformable Linear Objects with Offline and Online Learning of Unknown Models

DefGoalNet: Contextual Goal Learning from Demonstrations For Deformable Object Manipulation

Constraining Gaussian Process Implicit Surfaces for Robot Manipulation via Dataset Refinement

Modal-graph 3D shape servoing of deformable objects with raw point clouds

Gaussian process-based nonlinear predictive control for visual servoing of constrained mobile robots with unknown dynamics

Controlled Gaussian Process Dynamical Models with Application to Robotic Cloth Manipulation

Contour Moments Based Manipulation of Composite Rigid-Deformable Objects with Finite Time Model Estimation and Shape/Position Control

DeformerNet: Learning Bimanual Manipulation of 3D Deformable Objects

FBG-Based Online Learning and 3-D Shape Control of Unmodeled Continuum and Soft Robots in Unstructured Environments

Real-Time Deep Learning Approach to Visual Servo Control and Grasp Detection for Autonomous Robotic Manipulation

Model predictive manipulation of compliant objects with multi-objective optimizer and adversarial network for occlusion compensation

Optimal Cosserat-based deformation control for robotic manipulation of linear objects

Learning Robotic Manipulation through Visual Planning and Acting