Different gait combinations based on multi-modal deep CNN architectures

Büşranur Yaprak,Eyüp Gedikli
DOI: https://doi.org/10.1007/s11042-024-18859-9
IF: 2.577
2024-03-14
Multimedia Tools and Applications
Abstract:Gait recognition is the process of identifying a person from a distance based on their walking patterns. However, the recognition rate drops significantly under cross-view angle and appearance-based variations. In this study, the effectiveness of the most well-known gait representations in solving this problem is investigated based on deep learning. For this purpose, a comprehensive performance evaluation is performed by combining different modalities, including silhouettes, optical flows, and concatenated image of the Gait Energy Image (GEI) head and leg region, with GEI itself. This evaluation is carried out across different multimodal deep convolutional neural network (CNN) architectures, namely fine-tuned EfficientNet-B0, MobileNet-V1, and ConvNeXt-base models. These models are trained separately on GEIs, silhouettes, optical flows, and concatenated image of GEI head and leg regions, and then extracted GEI features are fused in pairs with other extracted modality features to find the most effective gait combination. Experimental results on the two different datasets CASIA-B and Outdoor-Gait show that the concatenated image of GEI head and leg regions significantly increased the recognition rate of the networks compared to other modalities. Moreover, this modality demonstrates greater robustness under varied carrying (BG) and clothing (CL) conditions compared to optical flows (OF) and silhouettes (SF). Codes available at https://github.com/busrakckugurlu/Different-gait-combinations-based-on-multi-modal-deep-CNN-architectures.git
computer science, information systems, theory & methods,engineering, electrical & electronic, software engineering
What problem does this paper attempt to address?
### The Problem the Paper Aims to Solve The paper aims to address the issue of performance degradation in gait recognition under different cross-view angles and appearance-based variations. Specifically, gait recognition faces the following challenges in practical applications: 1. **Cross-view angle variations**: Gait patterns differ significantly under different viewing angles, leading to a notable decrease in recognition rates. 2. **Appearance variations**: Gait recognition performance is poor under conditions such as carrying items (e.g., backpacks) and changes in clothing (e.g., coats). To tackle these challenges, the paper investigates the effectiveness of various gait representation methods within a deep learning framework and improves recognition rates through the combination of multimodal deep convolutional neural network (CNN) architectures. The specific methods include: - **Multimodal data**: Combining silhouette images, optical flow images, and concatenated images of the head and leg regions of gait energy images (GEI). - **Multimodal CNN architecture**: Using pre-trained EfficientNet-B0, MobileNet-V1, and ConvNeXt-base models for feature extraction and fusion. Through experimental validation, the paper aims to find the most effective gait combination method to enhance the robustness and accuracy of gait recognition under different viewing angles and appearance variations.