Abstract:Existing methods combining skeleton and silhouette representations demonstrate explicit effectiveness for gait recognition. However, current related methods simply combine the video-level representations of model-based skeleton data and gait silhouettes for retrieval. Therefore, diverse skeleton information is not fully exploited in existing related works: Firstly, the position and movement of bones are not clear from individual silhouettes. This indicates that the frame-level interaction between features of skeletons and silhouettes is critical, which is ignored by previous methods. Secondly, diverse part-level skeleton-guided gait features are not fully captured in existing related approaches. To solve the above issues, we present a novel framework with multi-level skeleton-guided refinement, including frame-level, part-level, and video-level skeleton-guided refinement, for comprehensive skeleton-aided gait representation learning. First, two modules are proposed for frame-level skeleton-guided refinement. Specifically, Visual Skeleton Enhanced Backbone (VSEB) is proposed to visually highlight the global and part-level skeleton regions for the feature of each silhouette frame. Moreover, Cross-Visual-Model Frame-level Interaction (CVMFI) is proposed to further transfer the model-based skeleton information to features of the visual modalities. Secondly, part-level visual and model-based skeleton features are utilized to refine the final gait representation. Concretely, in VSEB, Part Skeleton Enhance Network (PSEN) is proposed to visually enhance the position and movement of part-level skeletons. In addition, Semantic Part Pooling (SPP) is proposed for capturing the model-based skeleton features of different semantic parts. Finally, as the video-level skeleton-guided refinement, multimodal video-level features are combined to boost the final recognition performance. Extensive experimental results on prevailing datasets demonstrate that our approach outperforms most existing methods, including the skeleton-aided multi-modal methods. With the multi-level refinement guided by the skeleton modalities, the framework is expected to provide a deeper understanding of skeleton-aided gait recognition.

Gait recognition based on 3D human body reconstruction and multi-granular feature fusion

Hybrid Silhouette-skeleton Body Representation for Gait Recognition

GaitMA: Pose-guided Multi-modal Feature Fusion for Gait Recognition

Gait Recognition Using Multichannel Convolution Neural Networks

Human Gait Recognition Based on Frame-by-Frame Gait Energy Images and Convolutional Long Short-Term Memory

Human Identification Based on Gait Manifold

Human Gait Recognition Based on Frontal-View Walking Sequences Using Multi-modal Feature Representations and Learning

Learning Rich Features for Gait Recognition by Integrating Skeletons and Silhouettes

Cross-view Gait Recognition Through Ensemble Learning

GaitRA: triple-branch multimodal gait recognition with larger effective receptive fields and mixed attention

TriGait: Aligning and Fusing Skeleton and Silhouette Gait Data via a Tri-Branch Network

Non-local Gait Feature Extraction and Human Identification

GaitGS: Temporal Feature Learning in Granularity and Span Dimension for Gait Recognition

GaitAMR: Cross-view gait recognition via aggregated multi-feature representation

GaitCTCG: cross-view gait recognition via cascaded residual temporal shift and comprehensive multi-granularity learning

Gait Recognition With Multi-Level Skeleton-Guided Refinement

JointsGait:A model-based Gait Recognition Method based on Gait Graph Convolutional Networks and Joints Relationship Pyramid Mapping

GaitSet: Cross-view Gait Recognition through Utilizing Gait as a Deep Set

A Multi-Stage Adaptive Feature Fusion Neural Network for Multimodal Gait Recognition

Robust Gait Recognition based on Deep CNNs with Camera and Radar Sensor Fusion

GaitRef: Gait Recognition with Refined Sequential Skeletons