Abstract:Self-supervised representation learning has proven constructive for skeleton-based action recognition. For better performance, existing methods mainly focus on (1) multi-modal data augmentations and (2) triplet contrastive samples construction. However, designing these strategies is always heuristics and hard. Instead of exploring more similar strategies, this paper addresses this issue with a different view and proposes a novel Contrastive Spatio-Temporal Clustering (CSTC) module. CSTC constructs a supervised signal (pseudo-label) of action sequences in an online clustering manner, and it is complementary to the recent data augmentations or triplet contrastive samples construction strategies. Specifically, CSTC can be formulated as an optimal transport problem. we introduce the spatio-temporal regularizations into the original optimal transport term to guide the pseudo-label generation, i.e., a semantic regularization learned by frame index is proposed to constrain the frame order, and a prior normal distribution regularization based on sampling characteristics of samples is proposed to maintain the dependability of spatial cluster assignments. Furthermore, to enhance the learning of latent features, we propose a Bidirectional Cross-modal Clustering Consistency Objective (B3CO) to enforce cluster assignments consistency for different modalities of the same sample. Last, since fusing spatial and temporal clustering losses directly during back-propagation will confuse the learned dimension-specific semantics, we propose a simple yet effective training strategy to fix it by training the model using these two losses alternately. By integrating the above designs into the MoCo framework, we propose a Contrastive Spatio-Temporal Clustering Network (CSTCN), which can excavate cross-modal discriminative spatio-temporal features in the clustering space. Experimental results on NTU RGB+D 60, NTU RGB+D 120, and PKU-MMD II datasets show that CSTCN achieves state of-the-art performance in both single- and multi-modal models, especially in the KNN and semi-supervised evaluation protocols. Besides, the key module CSTC shows good generalization capability, and achieves consistent performance improvement on the basis of several state-of-the-art methods which focus on data augmentations and triplet contrastive samples construction.

Augmented Skeleton Based Contrastive Action Learning with Momentum LSTM for Unsupervised Action Recognition

Contrastive Learning from Extremely Augmented Skeleton Sequences for Self-Supervised Action Recognition

Skeleton-Contrastive 3D Action Representation Learning

Self-Supervised Action Representation Learning Based on Asymmetric Skeleton Data Augmentation

Improving Self-Supervised Action Recognition from Extremely Augmented Skeleton Sequences

Contrast-reconstruction Representation Learning for Self-supervised Skeleton-based Action Recognition

Efficient Spatio-Temporal Contrastive Learning for Skeleton-Based 3-D Action Recognition

Self-Supervised 3D Skeleton Representation Learning with Active Sampling and Adaptive Relabeling for Action Recognition

Efficient Spatio-Temporal Contrastive Learning for Skeleton-Based 3D Action Recognition

Contrastive Learning from Spatio-Temporal Mixed Skeleton Sequences for Self-Supervised Skeleton-Based Action Recognition

Learning Representations by Contrastive Spatio-temporal Clustering for Skeleton-based Action Recognition

Part Aware Contrastive Learning for Self-Supervised Action Recognition

Actionlet-Dependent Contrastive Learning for Unsupervised Skeleton-Based Action Recognition

Hierarchical Consistent Contrastive Learning for Skeleton-Based Action Recognition with Growing Augmentations

HaLP: Hallucinating Latent Positives for Skeleton-based Self-Supervised Learning of Actions

Representation modeling learning with multi-domain decoupling for unsupervised skeleton-based action recognition

X-Invariant Contrastive Augmentation and Representation Learning for Semi-Supervised Skeleton-Based Action Recognition.

Multi-Granularity Anchor-Contrastive Representation Learning for Semi-Supervised Skeleton-Based Action Recognition

An Attention Enhanced Graph Convolutional LSTM Network for Skeleton-Based Action Recognition

Action Recognition Based on Global Optimal Similarity Measuring

Shifting Perspective to See Difference: A Novel Multi-View Method for Skeleton Based Action Recognition