Abstract:Reliably planning fingertip grasps for multi-fingered hands lies as a key challenge for many tasks including tool use, insertion, and dexterous in-hand manipulation. This task becomes even more difficult when the robot lacks an accurate model of the object to be grasped. Tactile sensing offers a promising approach to account for uncertainties in object shape. However, current robotic hands tend to lack full tactile coverage. As such, a problem arises of how to plan and execute grasps for multi-fingered hands such that contact is made with the area covered by the tactile sensors. To address this issue, we propose an approach to grasp planning that explicitly reasons about where the fingertips should contact the estimated object surface while maximizing the probability of grasp success. Key to our method's success is the use of visual surface estimation for initial planning to encode the contact constraint. The robot then executes this plan using a tactile-feedback controller that enables the robot to adapt to online estimates of the object's surface to correct for errors in the initial plan. Importantly, the robot never explicitly integrates object pose or surface estimates between visual and tactile sensing, instead it uses the two modalities in complementary ways. Vision guides the robots motion prior to contact; touch updates the plan when contact occurs differently than predicted from vision. We show that our method successfully synthesises and executes precision grasps for previously unseen objects using surface estimates from a single camera view. Further, our approach outperforms a state of the art multi-fingered grasp planner, while also beating several baselines we propose.

3-D Object Reconstruction Using Tactile Sensing for Multi-fingered Hand

Unknown Objects Grasp Planning Using Force/Torque Measurements

Sliding Touch-based Exploration for Modeling Unknown Object Shape with Multi-fingered Hands

In-Hand 3D Object Reconstruction from a Monocular RGB Video

Grasping Unknown Objects Based on 3d Model Reconstruction

Development of Flexible Tactile Sensor for the Envelop of Curved Robotic Hand Finger in Grasping Force Sensing

3D Dense Reconstruction of Vision-based Tactile Sensor with Coded Markers

9DTact: A Compact Vision-Based Tactile Sensor for Accurate 3D Shape Reconstruction and Generalizable 6D Force Estimation

Development of Robotic Hand Tactile Sensing System for Distributed Contact Force Sensing in Robotic Dexterous Multimodal Grasping

Dexterous object manipulation by a multi-fingered robotic hand with visual-tactile fingertip sensors

Model Recovery Of Unknown Objects From Discrete Tactile Points

Dynamic Modeling of Hand-Object Interactions via Tactile Sensing

Model-Based 3D Contact Geometry Perception for Visual Tactile Sensor

Development of a Three-Fingered Multi-Modality Dexterous Hand with Integrated Embedded High-Dimensional Sensors

A 3-D Surface Reconstruction with Shadow Processing for Optical Tactile Sensors

Visual-Tactile Sensing for In-Hand Object Reconstruction

Development of a Visual-Tactile Fingertip Sensor and an Object Manipulation Method using a Multi-Fingered Robotic Hand

Planning Visual-Tactile Precision Grasps via Complementary Use of Vision and Touch

Recognition of the Three-Dimensional Shape of Objects Grasped for Pesa Multi-Fingered Robot Hand

Vision-based Tactile Sensor Using Depth from Defocus for Artificial Finger in Hand Prosthesis

DTact: A Vision-Based Tactile Sensor That Measures High-Resolution 3D Geometry Directly from Darkness