Abstract:Tactile sensing is significant for robotics since it can obtain physical contact information during manipulation. To capture multimodal contact information within a compact framework, we designed a novel sensor called ViTacTip, which seamlessly integrates both tactile and visual perception capabilities into a single, integrated sensor unit. ViTacTip features a transparent skin to capture fine features of objects during contact, which can be known as the see-through-skin mechanism. In the meantime, the biomimetic tips embedded in ViTacTip can amplify touch motions during tactile perception. For comparative analysis, we also fabricated a ViTac sensor devoid of biomimetic tips, as well as a TacTip sensor with opaque skin. Furthermore, we develop a Generative Adversarial Network (GAN)-based approach for modality switching between different perception modes, effectively alternating the emphasis between vision and tactile perception modes. We conducted a performance evaluation of the proposed sensor across three distinct tasks: i) grating identification, ii) pose regression, and iii) contact localization and force estimation. In the grating identification task, ViTacTip demonstrated an accuracy of 99.72%, surpassing TacTip, which achieved 94.60%. It also exhibited superior performance in both pose and force estimation tasks with the minimum error of 0.08mm and 0.03N, respectively, in contrast to ViTac's 0.12mm and 0.15N. Results indicate that ViTacTip outperforms single-modality sensors.

What problem does this paper attempt to address?

The problem this paper attempts to address is that in robotics, a single perception modality (such as vision or touch) may not fully understand complex environments. Therefore, a multimodal sensor capable of capturing both visual and tactile information is needed. To this end, the paper designs and validates a novel bio-inspired physical vision-tactile fusion sensor named ViTacTip. Specifically, ViTacTip achieves seamless integration of visual and tactile perception capabilities by combining transparent skin and an embedded bio-inspired tactile tip within a compact framework. This design not only captures fine details of objects but also amplifies tactile motion, thereby improving perception accuracy. Additionally, the paper develops a method based on Generative Adversarial Networks (GAN) to switch between different perception modes, effectively balancing the focus of visual and tactile perception. To evaluate the performance of ViTacTip, the paper conducts tests on three tasks: 1. **Grating Recognition**: ViTacTip achieves an accuracy of 99.72% in the grating recognition task, surpassing TacTip's 94.60%. 2. **Pose Regression**: ViTacTip has the smallest error in the pose regression task, with 0.08 mm and 0.03 N, outperforming ViTac's 0.12 mm and 0.15 N. 3. **Contact Localization and Force Estimation**: ViTacTip performs excellently in the contact localization and force estimation task, accurately locating contact points under different shear conditions and lighting variations, and successfully predicting 3D forces. Overall, the paper aims to demonstrate the superior performance of ViTacTip in multimodal perception through its design and validation, particularly in applications such as precise grasping and dexterous manipulation.

ViTacTip: Design and Verification of a Novel Biomimetic Physical Vision-Tactile Fusion Sensor

M Tac: A Multispectral Multimodal Visuotactile Sensor With Beyond-Human Sensory Capabilities

M $^{3}$ Tac: A Multispectral Multimodal Visuotactile Sensor with Beyond-Human Sensory Capabilities

StereoTac: a Novel Visuotactile Sensor that Combines Tactile Sensing with 3D Vision

3D-ViTac: Learning Fine-Grained Manipulation with Visuo-Tactile Sensing

Multidimensional Tactile Sensor with a Thin Compound Eye-Inspired Imaging System

DelTact: A Vision-based Tactile Sensor Using Dense Color Pattern

Design, Optimization and Application of Tactile Sensor Based on Virtual Binocular Vision

Seeing Through your Skin: Recognizing Objects with a Novel Visuotactile Sensor

Interpreting and Predicting Tactile Signals for the SynTouch BioTac

MiniTac: An Ultra-Compact 8 mm Vision-Based Tactile Sensor for Enhanced Palpation in Robot-Assisted Minimally Invasive Surgery

PaLmTac: A Vision-based Tactile Sensor Leveraging Distributed-Modality Design and Modal-matching Recognition for Soft Hand Perception

SeeThruFinger: See and Grasp Anything with a Multi-Modal Soft Touch

GradTac: Spatio-Temporal Gradient Based Tactile Sensing

9DTact: A Compact Vision-Based Tactile Sensor for Accurate 3D Shape Reconstruction and Generalizable 6D Force Estimation

A Miniaturised Camera-based Multi-Modal Tactile Sensor

CompdVision: Combining Near-Field 3D Visual and Tactile Sensing Using a Compact Compound-Eye Imaging System

DTact: A Vision-Based Tactile Sensor That Measures High-Resolution 3D Geometry Directly from Darkness

A Vision-Based Tactile Sensing System for Multimodal Contact Information Perception via Neural Network

A Dual-Modal Vision-Based Tactile Sensor for Robotic Hand Grasping.

TIRgel: A Visuo-Tactile Sensor With Total Internal Reflection Mechanism for External Observation and Contact Detection