Abstract:In the field of gesture recognition, A-mode ultrasound (AUS) and surface electromyography (sEMG) exhibit distinct advantages and limitations. However, they synergistically compensate for each other’s drawbacks, thereby enhancing practical performance. To integrate these modalities effectively, we introduce MSMFNet, a novel multi-scale multi-modal fusion network. This network consists of two key stages: the feature extraction stage and the fusion stage. In the feature extraction stage, we utilize a dual-branch network design that is more suitable for heterogeneous data, allowing effective handling of different types of data and providing a richer feature representation for subsequent steps. Firstly, one branch constructs a multi-scale Conv-Transformer network. Utilizing a combination of convolutional neural networks (CNN) and Transformer, this branch extracts information at various scales from raw sEMG images, comprehensively capturing complex details and providing a global, multi-level feature representation for the task. The second branch utilizes multi-scale mixed convolution for feature extraction from stacked sEMG and AUS images, enabling simultaneous processing of information from two distinct sources. This facilitates the effective fusion of multi-modal information, enhancing the model’s overall understanding of the correlation between different data sources. In the fusion stage, we incorporate an innovative adaptive weight-learning fusion mechanism, departing from conventional fusion approaches. This design is crafted to forestall potential performance deterioration arising from feature conflicts. By dynamically learning weights, we guarantee a precise reflection of the importance of features extracted from diverse branches in the final outcome, thus significantly amplifying the network’s performance. MSMFNet achieves an average accuracy of 93.09% on our mixed dataset, surpassing current state-of-the-art multimodal fusion methods and validating its superiority and robustness.

A Multimodal Multistream Multilevel Fusion Network for Finger Joint Angle Estimation with Hybrid Semg and FMG Sensing

Continuous Motion Finger Joint Angle Estimation Utilizing Hybrid Semg-Fmg Modality Driven Transformer-Based Deep Learning Model

Multi-Attention Feature Fusion Network for Accurate Estimation of Finger Kinematics from Surface Electromyographic Signals

MSMFNet: Multi-Modal Fusion Gesture Recognition Network with Multi-Scale Integration of AUS and Semg

A Hybrid Multimodal Fusion Framework for Semg-Acc-based Hand Gesture Recognition

MSFF-Net: Multi-Stream Feature Fusion Network for surface electromyography gesture recognition

Long exposure convolutional memory network for accurate estimation of finger kinematics from surface electromyographic signals

Continuous Estimation of Human Joint Angles From sEMG Using a Multi-Feature Temporal Convolutional Attention-Based Network

Gesture Classification in Electromyography Signals for Real-Time Prosthetic Hand Control Using a Convolutional Neural Network-Enhanced Channel Attention Model

A Novel Multi-Feature Fusion Network With Spatial Partitioning Strategy and Cross-Attention for Armband-Based Gesture Recognition

Multi-View Fusion Network-Based Gesture Recognition Using sEMG Data

Finger Joint Angle Estimation Based on Motoneuron Discharge Activities

Multimodal Fusion Convolutional Neural Network Based on sEMG and Accelerometer Signals for Inter-Subject Upper Limb Movement Classification

An Evolutionary Network Architecture Search Framework with Adaptive Multimodal Fusion for Hand Gesture Recognition

AGMMLN: an Attention-Guided Multiscale Multitask Learning Network for Simultaneous Gesture and Force Level Recognition

Simultaneous Estimation of Hand Joints’ Angles Toward Semg-Driven Human–Robot Interaction

Finger Joint Angle Estimation based on sEMG signals and deep learning method

Multi-modal fusion for robust hand gesture recognition based on heterogeneous networks

A Multimodal Multilevel Converged Attention Network for Hand Gesture Recognition With Hybrid sEMG and A-Mode Ultrasound Sensing

Electromyography Based Cross-Subject Limb Angle Estimation via Hierarchical Spiking Attentional Feature Decomposition Network

Multimodal Human-Exoskeleton Interface for Lower Limb Movement Prediction Through a Dense Co-Attention Symmetric Mechanism.