Abstract:In general, robotic dexterous hands are equipped with various sensors for acquiring multimodal contact information such as position, force, and pose of the grasped object. This multi-sensor-based design adds complexity to the robotic system. In contrast, vision-based tactile sensors employ specialized optical designs to enable the extraction of tactile information across different modalities within a single system. Nonetheless, the decoupling design for different modalities in common systems is often independent. Therefore, as the dimensionality of tactile modalities increases, it poses more complex challenges in data processing and decoupling, thereby limiting its application to some extent. Here, we developed a multimodal sensing system based on a vision-based tactile sensor, which utilizes visual representations of tactile information to perceive the multimodal contact information of the grasped object. The visual representations contain extensive content that can be decoupled by a deep neural network to obtain multimodal contact information such as classification, position, posture, and force of the grasped object. The results show that the tactile sensing system can perceive multimodal tactile information using only one single sensor and without different data decoupling designs for different modal tactile information, which reduces the complexity of the tactile system and demonstrates the potential for multimodal tactile integration in various fields such as biomedicine, biology, and robotics.

Psychophysics of wearable haptic/tactile perception in a multisensory context

Unveiling interactions of spatial-temporal information in tactile motion perception

A Wearable Vision-To-Audio Sensory Substitution Device for Blind Assistance and the Correlated Neural Substrates

A multi-class tactile brain-computer interface based on stimulus-induced oscillatory dynamics.

Multi-modal Sensing-Based Interactive Glove System for Teleoperation and VR/AR

Cross-Modal Sensory Integration of Visual-Tactile Motion Information: Instrument Design and Human Psychophysics

A Skin-Integrated Multimodal Haptic Interface for Immersive Tactile Feedback

Tactile Sensing and Rendering Patch with Dynamic and Static Sensing and Haptic Feedback for Immersive Communication

Affective Haptics and Multimodal Experiments Research

A Vision-Based Tactile Sensing System for Multimodal Contact Information Perception via Neural Network

Recent Progress in Wearable Tactile Sensors Combined with Algorithms Based on Machine Learning and Signal Processing

Encoding of tactile information in hand via skin-integrated wireless haptic interface

Design, Optimization and Application of Tactile Sensor Based on Virtual Binocular Vision

Electronic Skins with Multimodal Sensing and Perception

Wearable Mechanotransduced Tactile Sensor for Haptic Perception

Simulating Tactile and Visual Multisensory Behaviour in Humans Based on an MoS2 Field Effect Transistor

Investigating the Effect of Sensory Concurrency on Learning Haptic Spatiotemporal Signals

Deciphering Contact Interactions and Exploration Strategies Underlying Tactile Perception of Material Softness

A bioinspired tactile scanner for computer haptics

A multi-dimensional tactile perception system based on triboelectric sensors: Towards intelligent sorting without seeing

Affective Haptics Research and Interaction Design.