Abstract:To learn from the numerous unlabeled data for smart infrastructure, we propose Enhanced Multi-Task Self-Supervised Learning (EMS2L) for self-supervised action recognition based on 3D human skeleton. With EMS2L, multiple self-supervised tasks are integrated to learn more comprehensive information, which is different from previous methods in which a single self-supervised task is manipulated. The self-supervised tasks employed here include task-specific methods (i.e., motion prediction and jigsaw puzzle task) and task-agnostic methods such as contrastive learning. Through the combination of these three self-supervised tasks, we can learn rich feature representations. Specifically, motion prediction is applied to extract detailed information by reconstructing original data from temporally masked and noisy sequences. Jigsaw puzzle makes the learned model capable of exploring temporal discriminative features for human action recognition by predicting the correct orders of shuffled sequences. Besides, to standardize the feature space, we utilize contrastive learning to constrain feature learning to increase the compactness within the class and separability between classes. To learn invariant representations, an attention model is proposed for contrastive representation learning to reduce the distance between original features and attention features. To avoid the performance degradation of network representation due to the pursuit of excessive invariance, this attention-based contrastive learning gives different degrees of weights to the features of different transformed data. Under a variety of settings, including fully-supervised, semi-supervised, unsupervised, and transfer learning, we evaluate EMS2L with downstream tasks. We also explore different network architectures (i.e., GRU GCN). The remarkable results on NW-UCLA, NTU RGB+D, and PKUMMD datasets illustrate the generality of our approach. With sufficient and extensive experiments, the advantage of our method is demonstrated by learning features that are more general and discriminative. Besides, we further provide more experimental analysis for different self-supervised tasks.

Discriminative Multi-instance Multitask Learning for 3D Action Recognition.

A 3D-CNN and LSTM Based Multi-Task Learning Architecture for Action Recognition.

Multi-Domain and Multi-Task Learning for Human Action Recognition

Multi-Modality Co-Learning for Efficient Skeleton-based Action Recognition

Attention-Based Multilevel Co-Occurrence Graph Convolutional LSTM for 3-D Action Recognition

MS<SUP>2</SUP>L: Multi-Task Self-Supervised Learning for Skeleton Based Action Recognition

Multisource Learning for Skeleton-Based Action Recognition Using Deep LSTM and CNN

Online Robust Action Recognition Based on a Hierarchical Model

Action Recognition Based on Global Optimal Similarity Measuring

An Attentional Spatial Temporal Graph Convolutional Network with Co-Occurrence Feature Learning for Action Recognition

3D Action Recognition Using Multi-Temporal Skeleton Visualization.

Fusing Shape and Motion Matrices for View Invariant Action Recognition Using 3D Skeletons

Temporal Cues Enhanced Multimodal Learning for Action Recognition in RGB-D Videos

Motion Complement and Temporal Multifocusing for Skeleton-Based Action Recognition

Skeleton-Based Mutually Assisted Interacted Object Localization and Human Action Recognition

Generic Enhanced Ensemble Learning with Multi-Level Kinematic Constraints for 3D Action Recognition

Spatiotemporal Multimodal Learning With 3D CNNs for Video Action Recognition

Learning Discriminative Activated Simplices for Action Recognition

EMS2L: Enhanced Multi-Task Self-Supervised Learning for 3D Skeleton Representation Learning

Deep spatiotemporal LSTM network with temporal pattern feature for 3D human action recognition

Learning Composite Latent Structures for 3D Human Action Representation and Recognition