Abstract:Although there are many wearable sensors that make the acquisition of multi-modality data easier, effective feature extraction and fusion of the data is still challenging for lower limb locomotion mode recognition. In this article, a novel neural network is proposed for accurate prediction of five common lower limb locomotion modes including level walking, ramp ascent, ramp descent, stair ascent, and stair descent. First, the encoder-decoder structure is employed to enrich the channel diversity for the separation of the useful patterns from combined patterns. Second, a self-attention based cross-modality interaction module is proposed, which enables bilateral information flow between two encoding paths to fully exploit the interdependencies and to find complementary information between modalities. Third, a multi-modality fusion module is designed where the complementary features are fused by a channel-wise weighted summation whose coefficients are learned end-to-end. A benchmark dataset is collected from 10 health subjects containing EMG and IMU signals and five locomotion modes. Extensive experiments are conducted on one publicly available dataset ENABL3S and one self-collected dataset. The results show that the proposed method outperforms the compared methods with higher classification accuracy. The proposed method achieves a classification accuracy of 98.25 $\%$ on ENABL3S dataset and 95.51 $\%$ on the self-collected dataset. Note to Practitioners —This article aims to solve the real challenges encountered when intelligent recognition algorithms are applied in wearable robots: how to effectively and efficiently fuse the multi-modality data for better decision-making. First, most existing methods directly concatenate the multi-modality data, which increases the data dimensionality and brings computational burden. Second, existing recognition neural networks continuously compress the feature size such that the discriminative patterns are submerged in the noise and thus difficult to be identified. This research decomposes the mixed input signals on the channel dimension such that the useful patterns can be separated. Moreover, this research employs self-attention mechanism to associate correlations between two modalities and use this correlation as a new feature for subsequent representation learning, generating new, compact, and complementary features for classification. We demonstrate that the proposed network achieves 98.25 $\%$ accuracy and 3.5 ms prediction time. We anticipate that the proposed network could be a general scientific and practical methodology of multi-modality signal fusion and feature learning for intelligent systems.

Multi-scale Learning for Multimodal Neurophysiological Signals: Gait Pattern Classification as an Example

Identification of Gait-Related Brain Activity Using Electroencephalographic Signals

Multimodal Gait Recognition for Neurodegenerative Diseases

Research on motion pattern recognition of exoskeleton robot based on multimodal machine learning model

Classification of Neurological Gait Disorders Using Multi-task Feature Learning

Multiple Kernel Based Region Importance Learning for Neural Classification of Gait States from EEG Signals

Human Gait Phases Recognition Based on Multi-Source Data Fusion and BILSTM Attention Neural Network

Multimodal Gait Abnormality Recognition Using a Convolutional Neural Network–Bidirectional Long Short-Term Memory (CNN-BiLSTM) Network Based on Multi-Sensor Data Fusion

Gait parameter fitting and adaptive enhancement based on cerebral blood oxygen information

Cross-Modality Self-Attention and Fusion-Based Neural Network for Lower Limb Locomotion Mode Recognition

Gait pattern recognition based on electroencephalogram signals with common spatial pattern and graph attention networks

A New EEG-based Paradigm for Classifying Intention of Compound-Limbs Movement

Fall-related Gait Pattern Recognition Based on Surface Electromyography Using a Hybrid Neural Network with Transfer Learning

Gait Phase Recognition Based on Air-pressure Mechanomyogram and Sensor Fusion

Boosting lower-limb motor imagery performance through an ensemble method for gait rehabilitation

A Novel Multimodal Human-Exoskeleton Interface Based on EEG and Semg Activity for Rehabilitation Training.

Human Body Mixed Motion Pattern Recognition Method Based On Multi-Source Feature Parameter Fusion

Multi-Modal Gait Recognition with Unidirectional Cross-modal Alignment

A Multi-Information Fusion Method for Gait Phase Classification in Lower Limb Rehabilitation Exoskeleton

GMSN: An efficient multi-scale feature extraction network for gait recognition

Multimodal Human-Exoskeleton Interface for Lower Limb Movement Prediction Through a Dense Co-Attention Symmetric Mechanism.