Research on Multimodal Fusion Recognition Method of Upper Limb Motion Patterns

Wendong Wang,Chengzhi Zhao,Xin Li,Zhi-Qiang Zhang,Xiaoqing Yuan,Hanhao Li

DOI: https://doi.org/10.1109/tim.2023.3289556

IF: 5.6

2023-07-07

IEEE Transactions on Instrumentation and Measurement

Abstract:In order to solve the problems of single movement pattern recognition information and low recognition accuracy of multijoint upper limb exoskeleton rehabilitation training, a multimodal information fusion method with human surface electromyography (sEMG) and electrocardiogram (ECG) was proposed, and an Inception-Sim model for upper limb motion pattern recognition was designed. Integrating the advantages of multimodal information, inspired by the convolutional neural network processing image classification problem, the original signal was converted into a Gramian angular summation/difference fields-histogram of oriented gradient (GASF/GADF-HOG) image based on the principle of Grameen angle superposition/difference field, and the directional gradient histogram feature of the GASF/GADF image was extracted. The Inception-Sim model was constructed based on the Inception V3 model, and the human motion pattern recognition was completed on the basis of the transfer learning network. VGG16, ResNet-50, and other backbone networks were selected as comparison models. The recognition accuracy of each motion pattern for all participants reaches up to 90%, which is better than that of the control model. The average iteration speed of the proposed Inception-Sim model improved by about 21% compared to the control model. The experimental results show that the proposed multimodal information fusion recognition method can improve the accuracy and iteration speed of the upper limb motion recognition mode and then improve the effect of upper limb rehabilitation training.

engineering, electrical & electronic,instruments & instrumentation

What problem does this paper attempt to address?

The paper primarily addresses the issues of single motion mode recognition information and relatively low recognition accuracy in upper limb exoskeleton rehabilitation training. It proposes a multimodal information fusion method based on surface electromyography (sEMG) and electrocardiography (ECG), and designs an upper limb motion mode recognition model named Inception-Sim. Specifically, the research problems include: 1. **Improving the accuracy and robustness of motion mode recognition**: Enhancing the accuracy and robustness of motion mode recognition by fusing sEMG signals and ECG signals. 2. **Addressing the limitations of single-signal recognition**: Traditional upper limb motion mode recognition often relies solely on a single signal (such as sEMG), which limits its performance in practical applications. 3. **Optimizing computational efficiency**: Reducing the computational burden and improving the iteration speed of the model while ensuring recognition accuracy. To solve these problems, the authors propose the following key technologies and methods: - **Multimodal information fusion**: Using sEMG signals to represent the spatial motion direction of muscles and ECG signals to represent the dimension of motion intensity. - **Feature extraction**: Employing Gramian Angular Summation Field/Difference Field-Histogram of Oriented Gradients (GASF/GADF-HOG) techniques to convert raw signals into image features. - **Deep learning model**: Constructing the Inception-Sim model based on the Inception V3 network for motion mode recognition. - **Experimental validation**: Demonstrating the effectiveness of the proposed method through comparisons with baseline models such as VGG16 and ResNet-50. Ultimately, the experimental results show that the proposed multimodal information fusion recognition method can significantly improve the recognition accuracy and iteration speed of upper limb motion modes, thereby enhancing the effectiveness of upper limb rehabilitation training.

Research on Multimodal Fusion Recognition Method of Upper Limb Motion Patterns

Human Motion Pattern Recognition and Feature Extraction: An Approach Using Multi-Information Fusion

Multimodal Fusion Convolutional Neural Network Based on sEMG and Accelerometer Signals for Inter-Subject Upper Limb Movement Classification

Enhanced Recognition of Amputated Wrist and Hand Movements by Deep Learning Method Using Multimodal Fusion of Electromyography and Electroencephalography

Multi-View Fusion Network-Based Gesture Recognition Using sEMG Data

Toward Hand Pattern Recognition in Assistive and Rehabilitation Robotics Using EMG and Kinematics

Improving Hand Gesture Recognition Robustness to Dynamic Posture Variations by Multimodal Deep Feature Fusion

A motion recognition model for upper-limb rehabilitation exercises

From Unimodal to Multimodal: improving sEMG-Based Pattern Recognition via deep generative models

Research on Upper Limb Action Intention Recognition Method Based on Fusion of Posture Information and Visual Information

Multimodal Sensor Motion Intention Recognition Based on Three-Dimensional Convolutional Neural Network Algorithm

Multi-modal fusion for robust hand gesture recognition based on heterogeneous networks

An enhanced ResNet deep learning method for multimodal signal-based locomotion intention recognition

A Hybrid Multimodal Fusion Framework for Semg-Acc-based Hand Gesture Recognition

EMG-Centered Multisensory Based Technologies for Pattern Recognition in Rehabilitation: State of the Art and Challenges

Multi-user motion recognition using sEMG via discriminative canonical correlation analysis and adaptive dimensionality reduction

Sensor Fusion for Myoelectric Control Based on Deep Learning With Recurrent Convolutional Neural Networks

Transferable non-invasive modal fusion-transformer (NIMFT) for end-to-end hand gesture recognition

Improved Transfer Learning for Detecting Upper-Limb Movement Intention Using Mechanical Sensors in an Exoskeletal Rehabilitation System

Improving sEMG-based motion intention recognition for upper-limb amputees using transfer learning