Abstract:Self-supervised learning has demonstrated remarkable capability in representation learning for skeleton-based action recognition. Existing methods mainly focus on applying global data augmentation to generate different views of the skeleton sequence for contrastive learning. However, due to the rich action clues in the skeleton sequences, existing methods may only take a global perspective to learn to discriminate different skeletons without thoroughly leveraging the local relationship between different skeleton joints and video frames, which is essential for real-world applications. In this work, we propose a Partial Spatio-Temporal Learning (PSTL) framework to exploit the local relationship from a partial skeleton sequences built by a unique spatio-temporal masking strategy. Specifically, we construct a negative-sample-free triplet steam structure that is composed of an anchor stream without any masking, a spatial masking stream with Central Spatial Masking (CSM), and a temporal masking stream with Motion Attention Temporal Masking (MATM). The feature cross-correlation matrix is measured between the anchor stream and the other two masking streams, respectively. (1) Central Spatial Masking discards selected joints from the feature calculation process, where the joints with a higher degree of centrality have a higher possibility of being selected. (2) Motion Attention Temporal Masking leverages the motion of action and remove frames that move faster with a higher possibility. Our method achieves SOTA performance on NTU-60, NTU-120 and PKU-MMD under various downstream tasks. A practical evaluation is performed where some skeleton joints are lost in downstream tasks. In contrast to previous methods that suffer from large performance drops, our PSTL can still achieve remarkable results, validating the robustness of our method. Code: <a class="link-external link-https" href="https://github.com/YujieOuO/PSTL.git" rel="external noopener nofollow">this https URL</a>.

Segmentation and Completion of Human Motion Sequence via Temporal Learning of Subspace Variety Model

Temporal Constrained Feasible Subspace Learning for Human Pose Forecasting

Spatio-temporal LTSA and Its Application to Motion Decomposition

Revisiting the Temporal Modeling in Spatio-Temporal Predictive Learning under A Unified View

A Spatio-Temporal Transformer Network for Human Motion Prediction in Human-Robot Collaboration

Spatiotemporal Consistency Learning from Momentum Cues for Human Motion Prediction

A Hierarchical Spatio-Temporal Model for Human Activity Recognition.

Multi-motion Segmentation Via Co-attention-induced Heterogeneous Model Fitting

Spatio-temporal Manifold Learning for Human Motions via Long-horizon Modeling

Temporal Continual Learning with Prior Compensation for Human Motion Prediction

A Spatial-Temporal Topic Segmentation Model For Human Mobile Behavior

Sequential Subspace Clustering via Temporal Smoothness

Learning Clustering for Motion Segmentation

Self-supervised Action Representation Learning from Partial Spatio-Temporal Skeleton Sequences

Multi-task learning for subspace segmentation

Machine Learning for Spatiotemporal Sequence Forecasting: A Survey

Is Single Enough? A Joint Spatiotemporal Feature Learning Framework for Multivariate Time Series Prediction

Multiscale Spatial and Temporal Learning for Human Motion Prediction

Class-guided Human Motion Prediction Via Multi-Spatial-temporal Supervision

Learning Human Motion from Monocular Videos via Cross-Modal Manifold Alignment

Spatial Temporal Pyramid Matching Using Temporal Sparse Representation for Human Motion Retrieval