Abstract:Skeleton-based action recognition, as a crucial research direction in computer vision, confronts numerous issues and challenges. Most existing research methods rely heavily on extensive labeled data for training, which significantly constraints their training effectiveness and generalization capability when labeled data is scarce. Consequently, how to integrate labeled and unlabeled data to overcome the limitations imposed by label scarcity has emerged as a pivotal research focus in skeleton-based action recognition. Targeting this label scarcity problem, this paper introduces a semi-supervised skeleton-based action recognition approach leveraging multi-dimensional feature-based graph contrastive learning. Firstly, three feature extractors are devised to extract and exploit the available informative cues from limited data thoroughly. The holistic feature extractor comprises five spatio-temporal graph convolutional blocks and a global average pooling layer. The detailed feature extractor is constructed by stacking the same spatio-temporal graph convolutional blocks, while the relational feature extractor primarily integrates stacked attention graph convolutional blocks and a global average pooling layer. Secondly, the sample relationship construction mechanism in graph contrastive learning is enhanced. A clustering process is employed to formulate soft positive/negative sample pairs based on sample similarity, and a sample connectivity matrix further weights the distances between these pairs, thereby enhancing classification accuracy. Furthermore, a novel loss function grounded in the information bottleneck theory is formulated to guide the model towards learning more robust and efficient skeleton action representations. Experimental evaluations demonstrate the superiority of our proposed method (MDKS) on two datasets, NTU60 and NW-UCLA. Specifically, on the NTU60 dataset, MDKS achieves classification accuracy improvements of 4.7% and 1.9% under the X-sub and X-view evaluation protocols, respectively, compared to the benchmark MAC-Learning method. On the NW-UCLA dataset, MDKS outperforms MAC-Learning by 1.4%, 1.2%, 1.9%, and 1.4% in classification accuracy under different labeled data ratios ranging from 5% to 40%. This work offers novel insights and methodologies for advancing skeleton-based action recognition. Future research will delve into label imbalance, label noise, multi-modal information fusion, and cross-scene generalization capabilities.

See Your Emotion from Gait Using Unlabeled Skeleton Data.

Self-supervised Gait-based Emotion Representation Learning from Selective Strongly Augmented Skeleton Sequences

A Multi-Head Pseudo Nodes Based Spatial–temporal Graph Convolutional Network for Emotion Perception from GAIT

G-GCSN: Global Graph Convolution Shrinkage Network for Emotion Perception from Gait

Take an Emotion Walk: Perceiving Emotions from Gaits Using Hierarchical Attention Pooling and Affective Mapping

Data Augmentation by Separating Identity and Emotion Representations for Emotional Gait Recognition.

Human Gait Recognition Based on Self-Adaptive Hidden Markov Model

Contrastive 3D Human Skeleton Action Representation Learning via CrossMoCo With Spatiotemporal Occlusion Mask Data Augmentation.

Skeleton-Contrastive 3D Action Representation Learning

Condition-Adaptive Graph Convolution Learning for Skeleton-Based Gait Recognition

SkeletonGait: Gait Recognition Using Skeleton Maps

Looking into Gait for Perceiving Emotions via Bilateral Posture and Movement Graph Convolutional Networks

ST-Gait++: Leveraging spatio-temporal convolutions for gait-based emotion recognition on videos

Multi-task Learning for Gait-Based Identity Recognition and Emotion Recognition Using Attention Enhanced Temporal Graph Convolutional Network.

Augmented Skeleton Based Contrastive Action Learning with Momentum LSTM for Unsupervised Action Recognition

Contrastive Learning from Extremely Augmented Skeleton Sequences for Self-Supervised Action Recognition

Condition-Aware Comparison Scheme for Gait Recognition

Semi-supervised learning for skeleton behavior recognition: A multi-dimensional graph comparison approach

Context-Sensitive Temporal Feature Learning for Gait Recognition

MetaGait: Learning to Learn an Omni Sample Adaptive Representation for Gait Recognition

Learning Representations by Contrastive Spatio-temporal Clustering for Skeleton-based Action Recognition