Abstract:The critical problem in skeleton-based action recognition is to extract high-level semantics from dynamic changes between skeleton joints. Therefore, Graph Convolutional Networks (GCNs) are widely applied to capture the spatial-temporal information of dynamic joint coordinates by graph-based convolution. However, previous GCNS with fixed graph convolution kernel are limited to the static topology of graphs and the geometric variations of actions. Moreover, the local information of adjacent nodes of the graph is aggregated layer by layer, which increases the model complexity. In this work, a Deformable Graph Convolutional Transformer (DGT) for skeleton-based action recognition is proposed to extract adaptive features via a flexible receptive field that is learnable. In our DGT model, a multiple-input-branches (MIB) architecture is adopted to obtain multiple information, such as joints, bones, and motions. The multiple features are fused in the Transformer Classifier. Then, the Spatial-Temporal Graph Convolution units (STGC) are used to learn a preliminary feature representation indicating both spatial and temporal dependencies on the graph. Next, a Deformable spatial-temporal compound attention backbone is followed, which learns to represent a robust feature via adaptive deformable skeleton features. The adaptive representation is obtained by dynamically adjusting its receptive field owing to the offset-based convolution method. In addition, a self-attention-based transformer classifier (TC) is designed to encode the sequence of features flattened on the spatial and temporal dimensions. The fully-connected attention mechanism further helps the high-level semantic representation by focusing on essential nodes in the graph. We evaluated DGT on two challenging large-scale datasets, NTU-RGBD 60 and NTU-RGBD 120. Experiment results support the efficacy of DGT to optimize the attention for different joints adaptively. A comparable performance but much more efficient than the state-of-the-art demonstrates the effectiveness of the proposed method.

Spatial-Temporal Synchronous Transformer for Skeleton-Based Hand Gesture Recognition

Decoupled Spatio-Temporal Grouping Transformer for Skeleton-Based Action Recognition

Spatio-Temporal Tuples Transformer for Skeleton-Based Action Recognition

A Local Spatial–Temporal Synchronous Network to Dynamic Gesture Recognition

STST: Spatial-Temporal Specialized Transformer for Skeleton-based Action Recognition

Spatial-temporal Transformer for Skeleton-based Action Recognition

Spatial-Temporal Unfold Transformer for Skeleton-based Human Action Recognition

STSD: spatial–temporal semantic decomposition transformer for skeleton-based action recognition

A Spatio-Temporal Transformer Network for Human Motion Prediction in Human-Robot Collaboration

STDM-transformer: Space-time dual multi-scale transformer network for skeleton-based action recognition

MSST-RT: Multi-Stream Spatial-Temporal Relative Transformer for Skeleton-Based Action Recognition

MVHANet: Multi-view Hierarchical Aggregation Network for Skeleton-Based Hand Gesture Recognition

Spatial Temporal Transformer Network for Skeleton-based Action Recognition

Hand-Transformer: Non-Autoregressive Structured Modeling for 3D Hand Pose Estimation

Spatial-Temporal Attention Res-TCN for Skeleton-Based Dynamic Hand Gesture Recognition

Skeleton-Based Gesture Recognition Using Several Fully Connected Layers with Path Signature Features and Temporal Transformer Module.

Deformable graph convolutional transformer for skeleton-based action recognition

Recognizing Skeleton-Based Hand Gestures by a Spatio-Temporal Network

A Convolutional-Transformer-Based Approach for Dynamic Gesture Recognition of Data Gloves

Pyramid Spatial-Temporal Graph Transformer for Skeleton-Based Action Recognition

STMR: Spiral Transformer for Hand Mesh Reconstruction