Abstract:Although there are many wearable sensors that make the acquisition of multi-modality data easier, effective feature extraction and fusion of the data is still challenging for lower limb locomotion mode recognition. In this article, a novel neural network is proposed for accurate prediction of five common lower limb locomotion modes including level walking, ramp ascent, ramp descent, stair ascent, and stair descent. First, the encoder-decoder structure is employed to enrich the channel diversity for the separation of the useful patterns from combined patterns. Second, a self-attention based cross-modality interaction module is proposed, which enables bilateral information flow between two encoding paths to fully exploit the interdependencies and to find complementary information between modalities. Third, a multi-modality fusion module is designed where the complementary features are fused by a channel-wise weighted summation whose coefficients are learned end-to-end. A benchmark dataset is collected from 10 health subjects containing EMG and IMU signals and five locomotion modes. Extensive experiments are conducted on one publicly available dataset ENABL3S and one self-collected dataset. The results show that the proposed method outperforms the compared methods with higher classification accuracy. The proposed method achieves a classification accuracy of 98.25 $\%$ on ENABL3S dataset and 95.51 $\%$ on the self-collected dataset. Note to Practitioners —This article aims to solve the real challenges encountered when intelligent recognition algorithms are applied in wearable robots: how to effectively and efficiently fuse the multi-modality data for better decision-making. First, most existing methods directly concatenate the multi-modality data, which increases the data dimensionality and brings computational burden. Second, existing recognition neural networks continuously compress the feature size such that the discriminative patterns are submerged in the noise and thus difficult to be identified. This research decomposes the mixed input signals on the channel dimension such that the useful patterns can be separated. Moreover, this research employs self-attention mechanism to associate correlations between two modalities and use this correlation as a new feature for subsequent representation learning, generating new, compact, and complementary features for classification. We demonstrate that the proposed network achieves 98.25 $\%$ accuracy and 3.5 ms prediction time. We anticipate that the proposed network could be a general scientific and practical methodology of multi-modality signal fusion and feature learning for intelligent systems.

An enhanced ResNet deep learning method for multimodal signal-based locomotion intention recognition

Cross-Modality Self-Attention and Fusion-Based Neural Network for Lower Limb Locomotion Mode Recognition

A SE-DenseNet-LSTM model for locomotion mode recognition in lower limb exoskeleton

Improved Transfer Learning for Detecting Upper-Limb Movement Intention Using Mechanical Sensors in an Exoskeletal Rehabilitation System

A Motion Intent Recognition Method for Lower Limbs Based on CNN-RF Combined Model

A Novel Method for Detecting Misclassifications of the Locomotion Mode in Lower-Limb Exoskeleton Robot Control

A Multimodal Framework Based on Deep Belief Network for Human Locomotion Intent Prediction

A Novel Fusion Strategy for Locomotion Activity Recognition Based on Multimodal Signals.

FSM-HSVM-Based Locomotion Mode Recognition for Exoskeleton Robot

Real-Time Continuous Locomotion Mode Recognition and Transition Prediction for Human With Lower Limb Exoskeleton

A Locomotion Intent Prediction System Based on Multi-Sensor Fusion.

Multimodal Human-Exoskeleton Interface for Lower Limb Movement Prediction Through a Dense Co-Attention Symmetric Mechanism.

Design and Control of a Lower Limb Rehabilitation Robot Based on Human Motion Intention Recognition with Multi-Source Sensor Information

A Novel Multimodal Human-Exoskeleton Interface Based on EEG and Semg Activity for Rehabilitation Training.

Real-time Onboard SVM-based Human Locomotion Recognition for a Bionic Knee Exoskeleton on Different Terrains

A CNN-Based Method for Intent Recognition Using Inertial Measurement Units and Intelligent Lower Limb Prosthesis.

Research on Multimodal Fusion Recognition Method of Upper Limb Motion Patterns

Integral Real-time Locomotion Mode Recognition Based on GA-CNN for Lower Limb Exoskeleton

Taking Locomotion Mode As Prior: One Algorithm-Enabled Gait Events and Kinematics Prediction on Various Terrains

Exoskeleton Motion Intention Recognition Based on Multi-sensor Information Fusion

Real-Time On-Board Recognition of Locomotion Modes for an Active Pelvis Orthosis