Abstract:Wearable robotic exoskeletons have emerged as an exciting new treatment tool for disorders affecting mobility; however, the human–machine interface, used by the patient for device control, requires further improvement before robotic assistance and rehabilitation can be widely adopted. One method, made possible through advancements in machine learning technology, is the use of bioelectrical signals, such as electroencephalography (EEG) and electromyography (EMG), to classify the user's actions and intentions. While classification using these signals has been demonstrated for many relevant control tasks, such as motion intention detection and gesture recognition, challenges in decoding the bioelectrical signals have caused researchers to seek methods for improving the accuracy of these models. One such method is the use of EEG–EMG fusion, creating a classification model that decodes information from both EEG and EMG signals simultaneously to increase the amount of available information. So far, EEG–EMG fusion has been implemented using traditional machine learning methods that rely on manual feature extraction; however, new machine learning methods have emerged that can automatically extract relevant information from a dataset, which may prove beneficial during EEG–EMG fusion. In this study, Convolutional Neural Network (CNN) models were developed using combined EEG–EMG inputs to determine if they have potential as a method of EEG–EMG fusion that automatically extracts relevant information from both signals simultaneously. EEG and EMG signals were recorded during elbow flexion–extension and used to develop CNN models based on time–frequency (spectrogram) and time (filtered signal) domain image inputs. The results show a mean accuracy of 80.51 ± 8.07% for a three-class output (33.33% chance level), with an F-score of 80.74%, using time–frequency domain-based models. This work demonstrates the viability of CNNs as a new method of EEG–EMG fusion and evaluates different signal representations to determine the best implementation of a combined EEG–EMG CNN. It leverages modern machine learning methods to advance EEG–EMG fusion, which will ultimately lead to improvements in the usability of wearable robotic exoskeletons.

Multimodal Fusion Convolutional Neural Network Based on sEMG and Accelerometer Signals for Inter-Subject Upper Limb Movement Classification

Research on Multimodal Fusion Recognition Method of Upper Limb Motion Patterns

Transferable non-invasive modal fusion-transformer (NIMFT) for end-to-end hand gesture recognition

Sensor Fusion for Myoelectric Control Based on Deep Learning With Recurrent Convolutional Neural Networks

Multi-Attention Feature Fusion Network for Accurate Estimation of Finger Kinematics from Surface Electromyographic Signals

Multi-modal fusion for robust hand gesture recognition based on heterogeneous networks

Multi-View Fusion Network-Based Gesture Recognition Using sEMG Data

A Hybrid Multimodal Fusion Framework for Semg-Acc-based Hand Gesture Recognition

A Multimodal Multilevel Converged Attention Network for Hand Gesture Recognition With Hybrid sEMG and A-Mode Ultrasound Sensing

Lower Limb Motion Recognition Based on sEMG and CNN-TL Fusion Model

A Novel Multi-Feature Fusion Network With Spatial Partitioning Strategy and Cross-Attention for Armband-Based Gesture Recognition

Gesture Classification in Electromyography Signals for Real-Time Prosthetic Hand Control Using a Convolutional Neural Network-Enhanced Channel Attention Model

MSFF-Net: Multi-Stream Feature Fusion Network for surface electromyography gesture recognition

Electroencephalogram and surface electromyogram fusion-based precise detection of lower limb voluntary movement using convolution neural network-long short-term memory model

Evaluating Convolutional Neural Networks as a Method of EEG–EMG Fusion

Convolutional Neural Network with Data Augmentation for Robust Myoelectric Control

Improving Hand Gesture Recognition Robustness to Dynamic Posture Variations by Multimodal Deep Feature Fusion

Multi-modality deep forest for hand motion recognition via fusing sEMG and acceleration signals

sEMG and IMU Data-Based Hand Gesture Recognition Method Using Multistream CNN With a Fine-Tuning Transfer Framework

Multimodal Multiresolution Data Fusion Using Convolutional Neural Networks for IoT Wearable Sensing

Motor Function Evaluation of Hemiplegic Upper-Extremities Using Data Fusion from Wearable Inertial and Surface EMG Sensors