Abstract:Abstract. In this era, intelligent vision computing has always been a fascinating field. With the rapid development in computer vision, dynamic gesture-based recognition systems have attracted significant attention. However, automatically recognizing skeleton-based human gestures in the form of sign language is complex and challenging. Most existing methods consider skeleton-based human gesture recognition as a standard video recognition problem, without considering the rich structure information among both joints and gesture frames. Graph convolutional networks (GCNs) are a promising way to leverage structure information to learn structure representations. However, adopting GCNs to tackle such gesture sequences both in spatial and temporal spaces is challenging as graph could be highly nonlinear and complex. To overcome this issue, we propose the spatiotemporal GCNs model to leverage the powerful spatiotemporal correlations to adaptively construct spatiotemporal graphs, called Aegles. Our method could dynamically attend to relatively significant spatiotemporal joints and construct different graphs, including spatial, temporal, and spatiotemporal graph, and well capturing the structure information in gesture sequences. Besides, we introduce the second-order information of the gesture skeleton data, i.e., the length and orientation of bones, to improve the representation of human hands and fingers. In addition, with the public sign language datasets, we use OpenPose technology to extract human gesture skeleton and obtain human skeleton video, building four skeleton-based sign language recognition datasets. Experimental results show that this Aegles outperforms the state-of-the-art ones and that the spatiotemporal correlations effectively boost the performance of human gesture recognition.

Gesture Recognition Matching Based on Dynamic Skeleton

Dynamic hand gesture recognition using hidden Markov models

Gesture Recognition Based on Skeletonization Algorithm and CNN with ASL Database

Human gesture recognition of dynamic skeleton using graph convolutional networks

Dynamic Hand Gesture Recognition Based On 3D Skeleton

Skeleton-Based Gesture Recognition Using Several Fully Connected Layers with Path Signature Features and Temporal Transformer Module.

Motion Feature Augmented Recurrent Neural Network for Skeleton-based Dynamic Hand Gesture Recognition

Temporal-Spatial Deformable Pose Network for Skeleton-Based Gesture Recognition

Recognizing Skeleton-Based Hand Gestures by a Spatio-Temporal Network

Real-Time Dynamic Gesture Recognition Method Based on Gaze Guidance

Spatial Temporal Graph Convolutional Networks for Skeleton-Based Dynamic Hand Gesture Recognition

Dynamic Gesture Recognition Research Using Kinect's Skeleton Information

A Local Spatial–Temporal Synchronous Network to Dynamic Gesture Recognition

MFA-Net: Motion Feature Augmented Network for Dynamic Hand Gesture Recognition from Skeletal Data

CNN+RNN Depth and Skeleton based Dynamic Hand Gesture Recognition

A Skeleton Object Detection-Based Dynamic Gesture Recognition Method

Skeleton Based Dynamic Hand Gesture Recognition using LSTM and CNN

Skeleton-based Dynamic Hand Gesture Recognition using 3D Depth Data

Skeleton Based Dynamic Hand Gesture Recognition using Short Term Sampling Neural Networks (STSNN).

Automatic 3D Skeleton-based Dynamic Hand Gesture Recognition Using Multi-Layer Convolutional LSTM.

Non-trajectory-based Gesture Recognition in Human-Computer Interaction Based on Hand Skeleton Data