Abstract:Modeling and manipulating elasto-plastic objects are essential capabilities for robots to perform complex industrial and household interaction tasks (e.g., stuffing dumplings, rolling sushi, and making pottery). However, due to the high degree of freedom of elasto-plastic objects, significant challenges exist in virtually every aspect of the robotic manipulation pipeline, e.g., representing the states, modeling the dynamics, and synthesizing the control signals. We propose to tackle these challenges by employing a particle-based representation for elasto-plastic objects in a model-based planning framework. Our system, RoboCraft, only assumes access to raw RGBD visual observations. It transforms the sensing data into particles and learns a particle-based dynamics model using graph neural networks (GNNs) to capture the structure of the underlying system. The learned model can then be coupled with model-predictive control (MPC) algorithms to plan the robot's behavior. We show through experiments that with just 10 minutes of real-world robotic interaction data, our robot can learn a dynamics model that can be used to synthesize control signals to deform elasto-plastic objects into various target shapes, including shapes that the robot has never encountered before. We perform systematic evaluations in both simulation and the real world to demonstrate the robot's manipulation capabilities and ability to generalize to a more complex action space, different tool shapes, and a mixture of motion modes. We also conduct comparisons between RoboCraft and untrained human subjects controlling the gripper to manipulate deformable objects in both simulation and the real world. Our learned model-based planning framework is comparable to and sometimes better than human subjects on the tested tasks.

RoboPack: Learning Tactile-Informed Dynamics Models for Dense Packing

Dexterous Manoeuvre Through Touch in a Cluttered Scene

Learning Robust Skills for Tightly Coordinated Arms in Contact-Rich Tasks

RoboCraft: Learning to See, Simulate, and Shape Elasto-Plastic Objects with Graph Networks

Tight Robot Packing in the Real World: A Complete Manipulation Pipeline with Robust Primitives

A Method of Tight Placement for Robotic Dense Packing

Towards reliable robot packing system based on deep reinforcement learning

Fast Model-Based Contact Patch and Pose Estimation for Highly Deformable Dense-Geometry Tactile Sensors

Proprioceptive and Exteroceptive Information Perception in a Fabric Soft Robotic Arm via Physical Reservoir Computing with minimal training data

Robust Learning of Tactile Force Estimation through Robot Interaction

Deep Dynamics Models for Learning Dexterous Manipulation

Dynamic-Resolution Model Learning for Object Pile Manipulation

Tactile Active Inference Reinforcement Learning for Efficient Robotic Manipulation Skill Acquisition

Contact-Implicit Model Predictive Control for Dexterous In-hand Manipulation: A Long-Horizon and Robust Approach

Polymorphic Robot Learning for Dynamic and Contact-Rich Handling of Soft-Rigid Objects.

Tactile-Based Insertion for Dense Box-Packing

Dynamic Modeling of Hand-Object Interactions via Tactile Sensing

Tactile-Morph Skills: Energy-Based Control Meets Data-Driven Learning

Learning Manipulation under Physics Constraints with Visual Perception

Learning Robotic Manipulation through Visual Planning and Acting

Learning thin deformable object manipulation with a multi-sensory integrated soft hand