Abstract:Although there are many wearable sensors that make the acquisition of multi-modality data easier, effective feature extraction and fusion of the data is still challenging for lower limb locomotion mode recognition. In this article, a novel neural network is proposed for accurate prediction of five common lower limb locomotion modes including level walking, ramp ascent, ramp descent, stair ascent, and stair descent. First, the encoder-decoder structure is employed to enrich the channel diversity for the separation of the useful patterns from combined patterns. Second, a self-attention based cross-modality interaction module is proposed, which enables bilateral information flow between two encoding paths to fully exploit the interdependencies and to find complementary information between modalities. Third, a multi-modality fusion module is designed where the complementary features are fused by a channel-wise weighted summation whose coefficients are learned end-to-end. A benchmark dataset is collected from 10 health subjects containing EMG and IMU signals and five locomotion modes. Extensive experiments are conducted on one publicly available dataset ENABL3S and one self-collected dataset. The results show that the proposed method outperforms the compared methods with higher classification accuracy. The proposed method achieves a classification accuracy of 98.25 $\%$ on ENABL3S dataset and 95.51 $\%$ on the self-collected dataset. Note to Practitioners —This article aims to solve the real challenges encountered when intelligent recognition algorithms are applied in wearable robots: how to effectively and efficiently fuse the multi-modality data for better decision-making. First, most existing methods directly concatenate the multi-modality data, which increases the data dimensionality and brings computational burden. Second, existing recognition neural networks continuously compress the feature size such that the discriminative patterns are submerged in the noise and thus difficult to be identified. This research decomposes the mixed input signals on the channel dimension such that the useful patterns can be separated. Moreover, this research employs self-attention mechanism to associate correlations between two modalities and use this correlation as a new feature for subsequent representation learning, generating new, compact, and complementary features for classification. We demonstrate that the proposed network achieves 98.25 $\%$ accuracy and 3.5 ms prediction time. We anticipate that the proposed network could be a general scientific and practical methodology of multi-modality signal fusion and feature learning for intelligent systems.

A CNN-Based Method for Intent Recognition Using Inertial Measurement Units and Intelligent Lower Limb Prosthesis.

Motion Intention Prediction and Joint Trajectories Generation Toward Lower Limb Prostheses Using EMG and IMU Signals

A Motion Intent Recognition Method for Lower Limbs Based on CNN-RF Combined Model

Motion Intent Recognition in Intelligent Lower Limb Prosthesis Using One-Dimensional Dual-Tree Complex Wavelet Transforms

Novel Feature Extraction and Locomotion Mode Classification Using Intelligent Lower-Limb Prosthesis

Gait-Event-Based human intention recognition approach for lower limb

Real-Time Continuous Locomotion Mode Recognition and Transition Prediction for Human With Lower Limb Exoskeleton

On-board Training Strategy for IMU-Based Real-Time Locomotion Recognition of Transtibial Amputees with Robotic Prostheses

Taking Locomotion Mode As Prior: One Algorithm-Enabled Gait Events and Kinematics Prediction on Various Terrains

An Evaluation of Hybrid Deep Learning Models for Classifying Multiple Lower Limb Actions

A Locomotion Intent Prediction System Based on Multi-Sensor Fusion.

Continuous Estimation of Human Knee Joint Angles by Fusing Kinematic and Myoelectric Signals

An Improved Extreme Learning Machine (ELM) Algorithm for Intent Recognition of Transfemoral Amputees With Powered Knee Prosthesis

Recognition of Upper Limb Action Intention Based on IMU

Cross-Modality Self-Attention and Fusion-Based Neural Network for Lower Limb Locomotion Mode Recognition

Improving Motion Intention Recognition for Trans-Radial Amputees Based on Semg and Transfer Learning

Improved Transfer Learning for Detecting Upper-Limb Movement Intention Using Mechanical Sensors in an Exoskeletal Rehabilitation System

Adaptive Locomotion Transition Recognition with Wearable Sensors for Lower Limb Robotic Prosthesis

Locomotion Prediction for Lower Limb Prostheses in Complex Environments Via Semg and Inertial Sensors

Multi-channel Separated Encoder Based Convolutional Neural Network for Locomotion Intention Recognition

An enhanced ResNet deep learning method for multimodal signal-based locomotion intention recognition