Abstract:Global aging leads to a surge in neurological diseases. Quantitative gait analysis for the early detection of neurological diseases can effectively reduce the impact of the diseases. Recently, extensive research has focused on gait-abnormality-recognition algorithms using a single type of portable sensor. However, these studies are limited by the sensor's type and the task specificity, constraining the widespread application of quantitative gait recognition. In this study, we propose a multimodal gait-abnormality-recognition framework based on a Convolutional Neural Network-Bidirectional Long Short-Term Memory (CNN-BiLSTM) network. The as-established framework effectively addresses the challenges arising from smooth data interference and lengthy time series by employing an adaptive sliding window technique. Then, we convert the time series into time–frequency plots to capture the characteristic variations in different abnormality gaits and achieve a unified representation of the multiple data types. This makes our signal processing method adaptable to several types of sensors. Additionally, we use a pre-trained Deep Convolutional Neural Network (DCNN) for feature extraction, and the consequently established CNN-BiLSTM network can achieve high-accuracy recognition by fusing and classifying the multi-sensor input data. To validate the proposed method, we conducted diversified experiments to recognize the gait abnormalities caused by different neuropathic diseases, such as amyotrophic lateral sclerosis (ALS), Parkinson's disease (PD), and Huntington's disease (HD). In the PDgait dataset, the framework achieved an accuracy of 98.89% in the classification of Parkinson's disease severity, surpassing DCLSTM's 96.71%. Moreover, the recognition accuracy of ALS, PD, and HD on the PDgait dataset was 100%, 96.97%, and 95.43% respectively, surpassing the majority of previously reported methods. These experimental results strongly demonstrate the potential of the proposed multimodal framework for gait abnormality identification. Due to the advantages of the framework, such as its suitability for different types of sensors and fewer training parameters, it is more suitable for gait monitoring in daily life and the customization of medical rehabilitation schedules, which will help more patients alleviate the harm caused by their diseases.

CJAM: Convolutional Neural Network Joint Attention Mechanism in Gait Recognition

Gait Recognition Using Multichannel Convolution Neural Networks

Human Gait Recognition Based on Frame-by-Frame Gait Energy Images and Convolutional Long Short-Term Memory

JointsGait:A model-based Gait Recognition Method based on Gait Graph Convolutional Networks and Joints Relationship Pyramid Mapping

Non-local Gait Feature Extraction and Human Identification

Three-dimensional convolutional neural networks applied to video sensor-based gait recognition

Novel Deep Learning Network for Gait Recognition Using Multimodal Inertial Sensors

Joint Metric Learning and Hierarchical Network for Gait Recognition

Robust Gait Recognition based on Deep CNNs with Camera and Radar Sensor Fusion

Wearable Device-Based Gait Recognition Using Angle Embedded Gait Dynamic Images and a Convolutional Neural Network

Gait-CNN-ViT: Multi-Model Gait Recognition with Convolutional Neural Networks and Vision Transformer

GaitCTCG: cross-view gait recognition via cascaded residual temporal shift and comprehensive multi-granularity learning

GaitMA: Pose-guided Multi-modal Feature Fusion for Gait Recognition

3D Local Convolutional Neural Networks for Gait Recognition.

GaitRA: triple-branch multimodal gait recognition with larger effective receptive fields and mixed attention

Attention-Based Network For Cross-View Gait Recognition

Multimodal Gait Abnormality Recognition Using a Convolutional Neural Network–Bidirectional Long Short-Term Memory (CNN-BiLSTM) Network Based on Multi-Sensor Data Fusion

A Comprehensive Study on Cross-View Gait Based Human Identification with Deep CNNs

GMSN: An efficient multi-scale feature extraction network for gait recognition

GLGait: A Global-Local Temporal Receptive Field Network for Gait Recognition in the Wild

Action Recognition Based on Joint Trajectory Maps Using Convolutional Neural Networks