Abstract:Humans rely on their visual and tactile senses to develop a comprehensive 3D understanding of their physical environment. Recently, there has been a growing interest in exploring and manipulating objects using data-driven approaches that utilise high-resolution vision-based tactile sensors. However, 3D shape reconstruction using tactile sensing has lagged behind visual shape reconstruction because of limitations in existing techniques, including the inability to generalise over unseen shapes, the absence of real-world testing, and limited expressive capacity imposed by discrete representations. To address these challenges, we propose TouchSDF, a Deep Learning approach for tactile 3D shape reconstruction that leverages the rich information provided by a vision-based tactile sensor and the expressivity of the implicit neural representation DeepSDF. Our technique consists of two components: (1) a Convolutional Neural Network that maps tactile images into local meshes representing the surface at the touch location, and (2) an implicit neural function that predicts a signed distance function to extract the desired 3D shape. This combination allows TouchSDF to reconstruct smooth and continuous 3D shapes from tactile inputs in simulation and real-world settings, opening up research avenues for robust 3D-aware representations and improved multimodal perception in robotics. Code and supplementary material are available at: <a class="link-external link-https" href="https://touchsdf.github.io/" rel="external noopener nofollow">this https URL</a>

3D Dense Reconstruction of Vision-based Tactile Sensor with Coded Markers

DenseTact: Optical Tactile Sensor for Dense Shape Reconstruction

9DTact: A Compact Vision-Based Tactile Sensor for Accurate 3D Shape Reconstruction and Generalizable 6D Force Estimation

DTact: A Vision-Based Tactile Sensor That Measures High-Resolution 3D Geometry Directly from Darkness

DTact: A Vision-Based Tactile Sensor that Measures High-Resolution 3D Geometry Directly from Darkness

A Stable and Efficient Vision-Based Tactile Sensor with Tactile Detection Using Neural Network.

Design, Optimization and Application of Tactile Sensor Based on Virtual Binocular Vision

Model-Based 3D Contact Geometry Perception for Visual Tactile Sensor

Real-Time Reconstruction of 3-D Tactile Motion Field via Multitask Learning

A 3-D Surface Reconstruction with Shadow Processing for Optical Tactile Sensors

DelTact: A Vision-based Tactile Sensor Using Dense Color Pattern

A High-Resolution, Ultrabroad-Range and Sensitive Capacitive Tactile Sensor Based on a CNT/PDMS Composite for Robotic Hands.

Acquisition and Prediction of High-Density Tactile Field Data for Rigid and Flexible Objects

StereoTac: a Novel Visuotactile Sensor that Combines Tactile Sensing with 3D Vision

Real-time Marker Recognition Using Vision-based Tactile Sensor

TouchSDF: A DeepSDF Approach for 3D Shape Reconstruction using Vision-Based Tactile Sensing

DenseTact 2.0: Optical Tactile Sensor for Shape and Force Reconstruction

Visual-Tactile Sensing for In-Hand Object Reconstruction

Vision-based Tactile Sensor Using Depth from Defocus for Artificial Finger in Hand Prosthesis

A Dual-Modal Vision-Based Tactile Sensor for Robotic Hand Grasping.

Vision-based Tactile Image Generation via Contact Condition-guided Diffusion Model