Abstract:3D Skeleton-based human action recognition has attracted increasing attention in recent years. Most of the existing work focuses on supervised learning which requires a large number of labeled action sequences that are often expensive and time-consuming to annotate. In this paper, we address self-supervised 3D action representation learning for skeleton-based action recognition. We investigate self-supervised representation learning and design a novel skeleton cloud colorization technique that is capable of learning spatial and temporal skeleton representations from unlabeled skeleton sequence data. We represent a skeleton action sequence as a 3D skeleton cloud and colorize each point in the cloud according to its temporal and spatial orders in the original (unannotated) skeleton sequence. Leveraging the colorized skeleton point cloud, we design an auto-encoder framework that can learn spatial-temporal features from the artificial color labels of skeleton joints effectively. Specifically, we design a two-steam pretraining network that leverages fine-grained and coarse-grained colorization to learn multi-scale spatial-temporal features. In addition, we design a Masked Skeleton Cloud Repainting task that can pretrain the designed auto-encoder framework to learn informative representations. We evaluate our skeleton cloud colorization approach with linear classifiers trained under different configurations, including unsupervised, semi-supervised, fully-supervised, and transfer learning settings. Extensive experiments on NTU RGB+D, NTU RGB+D 120, PKU-MMD, NW-UCLA, and UWA3D datasets show that the proposed method outperforms existing unsupervised and semi-supervised 3D action recognition methods by large margins and achieves competitive performance in supervised 3D action recognition as well.

SkeletonMAE: Spatial-Temporal Masked Autoencoders for Self-supervised Skeleton Action Recognition.

SkeletonMAE: Graph-based Masked Autoencoder for Skeleton Sequence Pre-training.

Skeleton Based Action Recognition using a Stacked Denoising Autoencoder with Constraints of Privileged Information

Spatial Hierarchy and Temporal Attention Guided Cross Masking for Self-supervised Skeleton-based Action Recognition

Skeleton2vec: A Self-supervised Learning Framework with Contextualized Target Representations for Skeleton Sequence

Self-supervised Video Representation Learning with Motion-Aware Masked Autoencoders

GD-MAE: Generative Decoder for MAE Pre-training on LiDAR Point Clouds

Masked Motion Predictors Are Strong 3D Action Representation Learners

AdaMAE: Adaptive Masking for Efficient Spatiotemporal Learning with Masked Autoencoders

SMAM: Self and Mutual Adaptive Matching for Skeleton-Based Few-Shot Action Recognition

Concatenated Masked Autoencoders as Spatial-Temporal Learner

Multi-Scale Adaptive Skeleton Transformer for action recognition

MaskCAE: Masked Convolutional AutoEncoder Via Sensor Data Reconstruction for Self-Supervised Human Activity Recognition

Masked Video and Body-worn IMU Autoencoder for Egocentric Action Recognition

Skeleton-based Attention-Aware Spatial-Temporal Model for Action Detection and Recognition.

Spatial-Temporal Data Augmentation Based on LSTM Autoencoder Network for Skeleton-Based Human Action Recognition

Self-Supervised 3D Action Representation Learning With Skeleton Cloud Colorization

Motion-Aware Mask Feature Reconstruction for Skeleton-Based Action Recognition

SA-DVAE: Improving Zero-Shot Skeleton-Based Action Recognition by Disentangled Variational Autoencoders

MS<SUP>2</SUP>L: Multi-Task Self-Supervised Learning for Skeleton Based Action Recognition