Abstract:This work proposes a novel multimodal tactile sensor utilizing triboelectric mechanism to simultaneously decouple three modalities: materials, curvatures, and pressure. Integrated with deep learning, an object recognition system can be achieved with high accuracy. It also can be further developed to detect object softness under different touch gestures, highlighting its potential for real‐world applications in robotics. Recent developments in robotics increasingly highlight the importance of sensing technology, especially tactile perception, in enabling robots to effectively engage with their environment and interpret physical interactions. Due to power efficiency and low cost, the triboelectric mechanism has been frequently studied for measuring pressure and identifying materials to enhance robot perception. Nevertheless, there has been limited exploration of using the triboelectric effect to detect curved surfaces, despite their prevalence in daily lives. Here, a triboelectric multimodal tactile sensor (TMTS) of multilayered structural design is proposed to recognize distinct materials, curvatures, and pressure simultaneously, thus decoupling different modalities to enable more accurate detection. By attaching sensors to robotic fingertips and leveraging deep learning analytics, the quantitative curvature measurement provides more precise insights into an object's detailed geometric characteristics rather than merely assessing its overall shape, hence achieving automatic recognition of 12 grasped objects with 99.2% accuracy. The sensor can be further used to accurately recognize the softness of objects under different touch gestures of a robotic hand, achieving a 94.1% accuracy, demonstrating its significant potential for wide‐ranging applications in a future robotic‐enabled intelligent society.

Multi-modal Haptic Image Recognition Based on Deep Learning

Magnetostrictive-based multimodal tactile sensors for object recognition

A Vision-Based Tactile Sensing System for Multimodal Contact Information Perception via Neural Network

Multi-modal learning for gesture recognition

TALON: Improving Large Language Model Cognition with Tactility-Vision Fusion

Augmented Tactile Perception of Robotic Fingers Enabled by AI‐Enhanced Triboelectric Multimodal Sensors

A Dual-Modal Vision-Based Tactile Sensor for Robotic Hand Grasping.

Surface Material Recognition Using Active Multi-modal Extreme Learning Machine.

Visual Attention and Haptic Control: A Cross-Study.

A Multiorder Attentional Spatial Interactive Convolutional Neural Network (Moas-Cnn) for Low-Resolution Haptic Recognition

Tactile Perception Information Recognition of Prosthetic Hand Based on DNN-LSTM

Skin‐inspired Multi‐Modal Mechanoreceptors for Dynamic Haptic Exploration

Multimodal Surface Sensing Based on Hybrid Flexible Triboelectric and Piezoresistive Sensor.

Object Recognition Using Tactile and Image Information.

Hierarchical Tactile Sensation Integration from Prosthetic Fingertips Enables Multi-Texture Surface Recognition

M Tac: A Multispectral Multimodal Visuotactile Sensor With Beyond-Human Sensory Capabilities

A multisensory-feedback tactile glove with dense coverage of sensing arrays for object recognition

Haptic Perception Of Unknown Object By Robot Hand: Exploration Strategy And Recognition Approach

Alignment and Multi-Scale Fusion for Visual-Tactile Object Recognition

Recognition of Tactile Attribute Strength and Category Using Convolutional Neural Network

Hybrid Connectionist Symbolic Model for Morphologic Recognition by Tactile Sensing