Abstract:Global aging leads to a surge in neurological diseases. Quantitative gait analysis for the early detection of neurological diseases can effectively reduce the impact of the diseases. Recently, extensive research has focused on gait-abnormality-recognition algorithms using a single type of portable sensor. However, these studies are limited by the sensor's type and the task specificity, constraining the widespread application of quantitative gait recognition. In this study, we propose a multimodal gait-abnormality-recognition framework based on a Convolutional Neural Network-Bidirectional Long Short-Term Memory (CNN-BiLSTM) network. The as-established framework effectively addresses the challenges arising from smooth data interference and lengthy time series by employing an adaptive sliding window technique. Then, we convert the time series into time–frequency plots to capture the characteristic variations in different abnormality gaits and achieve a unified representation of the multiple data types. This makes our signal processing method adaptable to several types of sensors. Additionally, we use a pre-trained Deep Convolutional Neural Network (DCNN) for feature extraction, and the consequently established CNN-BiLSTM network can achieve high-accuracy recognition by fusing and classifying the multi-sensor input data. To validate the proposed method, we conducted diversified experiments to recognize the gait abnormalities caused by different neuropathic diseases, such as amyotrophic lateral sclerosis (ALS), Parkinson's disease (PD), and Huntington's disease (HD). In the PDgait dataset, the framework achieved an accuracy of 98.89% in the classification of Parkinson's disease severity, surpassing DCLSTM's 96.71%. Moreover, the recognition accuracy of ALS, PD, and HD on the PDgait dataset was 100%, 96.97%, and 95.43% respectively, surpassing the majority of previously reported methods. These experimental results strongly demonstrate the potential of the proposed multimodal framework for gait abnormality identification. Due to the advantages of the framework, such as its suitability for different types of sensors and fewer training parameters, it is more suitable for gait monitoring in daily life and the customization of medical rehabilitation schedules, which will help more patients alleviate the harm caused by their diseases.

Gait Recognition based on Two-Stream CNNs with Multisensor Progressive Feature Fusion

Gait Recognition Using Multichannel Convolution Neural Networks

A Multi-Stage Adaptive Feature Fusion Neural Network for Multimodal Gait Recognition

GaitFFDA: Feature Fusion and Dual Attention Gait Recognition Model

GaitMA: Pose-guided Multi-modal Feature Fusion for Gait Recognition

AdaptiveGait: adaptive feature fusion network for gait recognition

Human Gait Recognition Based on Frame-by-Frame Gait Energy Images and Convolutional Long Short-Term Memory

Robust Gait Recognition based on Deep CNNs with Camera and Radar Sensor Fusion

A Fusion-Assisted Multi-Stream Deep Learning and ESO-Controlled Newton–Raphson-Based Feature Selection Approach for Human Gait Recognition

Cross-View Gait Recognition Based on Dual-Stream Network

Non-local Gait Feature Extraction and Human Identification

Novel Deep Learning Network for Gait Recognition Using Multimodal Inertial Sensors

GMSN: An efficient multi-scale feature extraction network for gait recognition

Multimodal Gait Abnormality Recognition Using a Convolutional Neural Network–Bidirectional Long Short-Term Memory (CNN-BiLSTM) Network Based on Multi-Sensor Data Fusion

Classification of Abnormal Gait Based on Improved Two-Stream CNN

Learning Rich Features for Gait Recognition by Integrating Skeletons and Silhouettes

Human Gait Recognition Based on Frontal-View Walking Sequences Using Multi-modal Feature Representations and Learning

Gait recognition with global–local feature fusion based on swin transformer-3DCNN

Gait recognition based on 3D human body reconstruction and multi-granular feature fusion

LiCAF: LiDAR-Camera Asymmetric Fusion for Gait Recognition

Gait Recognition Via Multiple Features and Views Information Fusion