Abstract:Gesture recognition based on surface electromyography (sEMG) has been widely used in the field of human-machine interaction (HMI). However, sEMG has limitations, such as low signal-to-noise ratio and insensitivity to fine finger movements, so we consider adding A-mode ultrasound (AUS) to enhance the recognition impact. To explore the influence of multisource sensing data on gesture recognition and better integrate the features of different modules. We proposed a multimodal multilevel converged attention network (MMCANet) model for multisource signals composed of sEMG and AUS. The proposed model extracts the hidden features of the AUS signal with a convolutional neural network (CNN). Meanwhile, a CNN-LSTM (long-short memory network) hybrid structure extracts some spatial-temporal features from the sEMG signal. Then, two types of CNN features from AUS and sEMG are spliced and transmitted to a transformer encoder to fuse the information and interact with sEMG features to produce hybrid features. Finally, the classification results are output employing fully connected layers. Attention mechanisms are used to adjust the weights of feature channels. We compared MMCANet's feature extraction and classification performance with that of manually extracted sEMG-AUS features using four traditional machine-learning (ML) algorithms. The recognition accuracy increased by at least 5.15%. In addition, we tried deep learning (DL) methods with CNN on single modals. The experimental results showed that the proposed model improved 14.31% and 3.80% over the CNN method with single sEMG and AUS, respectively. Compared with some state-of-the-art fusion techniques, our method also achieved better results.

Hybrid Connectionist Symbolic Model for Morphologic Recognition by Tactile Sensing

Multimodal Surface Sensing Based on Hybrid Flexible Triboelectric and Piezoresistive Sensor.

Magnetostrictive-based multimodal tactile sensors for object recognition

TALON: Improving Large Language Model Cognition with Tactility-Vision Fusion

Tactile Perception Information Recognition of Prosthetic Hand Based on DNN-LSTM

Recognition of surface texture with wearable tactile sensor array: A pilot Study

A Multifunctional Tactile Sensory System for Robotic Intelligent Identification and Manipulation Perception

A Multimodal Multilevel Converged Attention Network for Hand Gesture Recognition With Hybrid sEMG and A-Mode Ultrasound Sensing

A Human-Like Siamese-Based Visual-Tactile Fusion Model for Object Recognition

Gesture Classification in Electromyography Signals for Real-Time Prosthetic Hand Control Using a Convolutional Neural Network-Enhanced Channel Attention Model

Surface Texture Recognition by Deep Learning‐Enhanced Tactile Sensing

Enhancing Tactile Sensing in Robotics: Dual-Modal Force and Shape Perception with EIT-based Sensors and MM-CNN

Surface Recognition With a Tactile Finger Based on Automatic Features Transferred From Deep Learning

A Vision-Based Tactile Sensing System for Multimodal Contact Information Perception via Neural Network

A Hybrid CNN-LSTM Network for Hand Gesture Recognition with Surface EMG Signals

Texture Recognition Based on Perception Data from a Bionic Tactile Sensor

Bioinspired multisensory neural network with crossmodal integration and recognition

Pattern recognition of surface electromyography based on multi-scale convolutional neural network with attention mechanism.

Design a Multifunctional Soft Tactile Sensor Enhanced by Machine Learning Approaches

Multifunctional Tactile Sensors for Object Recognition

Object Recognition Using Tactile and Image Information.