GaitRA: triple-branch multimodal gait recognition with larger effective receptive fields and mixed attention

Leren Xue,Zhuo Tao

DOI: https://doi.org/10.1007/s11042-024-19596-9

IF: 2.577

2024-06-21

Multimedia Tools and Applications

Abstract:Gait Recognition, as a long-distance biometric technique for identity recognition, has attracted widespread attention in recent years. Previous academia typically employs minor convolutional networks to extract single-modal features from the silhouette or the joint skeleton. Nevertheless, silhouette-based methods are susceptible to clothing variations, while skeleton-based methods encounter the issue of missing physique information. Therefore, we propose a novel multimodal triple-branch network dubbed GaitRA to comprehensively acquire gait features from two aspects of silhouette and skeleton simultaneously. GaitRA consists of three branches: a 3D-CNN branch to extract the primary features of silhouette sequences, a 2D-CNN branch to obtain the secondary features of silhouette sequences, and a branch based on Spatio-Temporal Graph Convolution (ST-GCN) to gain joint skeleton features. More importantly, we innovatively introduce the RepLK-ACTION Module, which combines the RepLK Block based on the Swin Transformer and the ACTION Module based on a mixed attention mechanism from the Action Recognition. RepLK-ACTION Module establishes larger Effective Receptive Fields (ERFs) through the large-kernel re-parameterized convolution to attain more discriminative multimodal gait information, thereby enhancing the model performance under complex walking conditions. Experiments have demonstrated that GaitRA significantly improves performance on both CASIA-B and Gait3D datasets. Especially, the proposed method achieves excellent results of 73.1% (Rank-1), 86.4% (Rank-5), 64.3% (mAP), and 36.2% (mINP) on the Gait3D dataset.

computer science, information systems, theory & methods,engineering, electrical & electronic, software engineering