Abstract:Self-supervised learning has demonstrated remarkable capability in representation learning for skeleton-based action recognition. Existing methods mainly focus on applying global data augmentation to generate different views of the skeleton sequence for contrastive learning. However, due to the rich action clues in the skeleton sequences, existing methods may only take a global perspective to learn to discriminate different skeletons without thoroughly leveraging the local relationship between different skeleton joints and video frames, which is essential for real-world applications. In this work, we propose a Partial Spatio-Temporal Learning (PSTL) framework to exploit the local relationship from a partial skeleton sequences built by a unique spatio-temporal masking strategy. Specifically, we construct a negative-sample-free triplet steam structure that is composed of an anchor stream without any masking, a spatial masking stream with Central Spatial Masking (CSM), and a temporal masking stream with Motion Attention Temporal Masking (MATM). The feature cross-correlation matrix is measured between the anchor stream and the other two masking streams, respectively. (1) Central Spatial Masking discards selected joints from the feature calculation process, where the joints with a higher degree of centrality have a higher possibility of being selected. (2) Motion Attention Temporal Masking leverages the motion of action and remove frames that move faster with a higher possibility. Our method achieves SOTA performance on NTU-60, NTU-120 and PKU-MMD under various downstream tasks. A practical evaluation is performed where some skeleton joints are lost in downstream tasks. In contrast to previous methods that suffer from large performance drops, our PSTL can still achieve remarkable results, validating the robustness of our method. Code: <a class="link-external link-https" href="https://github.com/YujieOuO/PSTL.git" rel="external noopener nofollow">this https URL</a>.

Language-Guided 3-D Action Feature Learning Without Ground-Truth Sample Class Label

Beyond Pattern Variance: Unsupervised 3-D Action Representation Learning With Point Cloud Sequence

Self-Supervised 3D Action Representation Learning With Skeleton Cloud Colorization

Self-supervised Action Representation Learning from Partial Spatio-Temporal Skeleton Sequences

Language-Assisted 3D Scene Understanding

Improving Self-Supervised Action Recognition from Extremely Augmented Skeleton Sequences

Vision-Language Meets the Skeleton: Progressively Distillation with Cross-Modal Knowledge for 3D Action Representation Learning

TAMC: Textual Alignment and Masked Consistency for Open-Vocabulary 3D Scene Understanding

Efficient Spatio-Temporal Contrastive Learning for Skeleton-Based 3D Action Recognition

Joint Representation Learning for Text and 3D Point Cloud

Skeleton Cloud Colorization for Unsupervised 3D Action Representation Learning

Spatial-Temporal Asynchronous Normalization for Unsupervised 3D Action Representation Learning

Self-Supervised 3D Skeleton Representation Learning with Active Sampling and Adaptive Relabeling for Action Recognition

Actionlet-Dependent Contrastive Learning for Unsupervised Skeleton-Based Action Recognition

CrossGLG: LLM Guides One-shot Skeleton-based 3D Action Recognition in a Cross-level Manner

More Text, Less Point: Towards 3D Data-Efficient Point-Language Understanding

3D Human Action Representation Learning via Cross-View Consistency Pursuit

Language-Assisted Human Part Motion Learning for Skeleton-Based Temporal Action Segmentation

Generalized Label-Efficient 3D Scene Parsing via Hierarchical Feature Aligned Pre-Training and Region-Aware Fine-tuning

Language-Assisted 3D Feature Learning for Semantic Scene Understanding