Abstract:Abstract. In this era, intelligent vision computing has always been a fascinating field. With the rapid development in computer vision, dynamic gesture-based recognition systems have attracted significant attention. However, automatically recognizing skeleton-based human gestures in the form of sign language is complex and challenging. Most existing methods consider skeleton-based human gesture recognition as a standard video recognition problem, without considering the rich structure information among both joints and gesture frames. Graph convolutional networks (GCNs) are a promising way to leverage structure information to learn structure representations. However, adopting GCNs to tackle such gesture sequences both in spatial and temporal spaces is challenging as graph could be highly nonlinear and complex. To overcome this issue, we propose the spatiotemporal GCNs model to leverage the powerful spatiotemporal correlations to adaptively construct spatiotemporal graphs, called Aegles. Our method could dynamically attend to relatively significant spatiotemporal joints and construct different graphs, including spatial, temporal, and spatiotemporal graph, and well capturing the structure information in gesture sequences. Besides, we introduce the second-order information of the gesture skeleton data, i.e., the length and orientation of bones, to improve the representation of human hands and fingers. In addition, with the public sign language datasets, we use OpenPose technology to extract human gesture skeleton and obtain human skeleton video, building four skeleton-based sign language recognition datasets. Experimental results show that this Aegles outperforms the state-of-the-art ones and that the spatiotemporal correlations effectively boost the performance of human gesture recognition.

Gesture Recognition Based on Skeletonization Algorithm and CNN with ASL Database

A Recognition Method for Italian Alphabet Gestures Based on Convolutional Neural Network

Gesture Recognition Matching Based on Dynamic Skeleton

A Recognition Method of Hand Gesture with CNN-SVM Model.

Gesture Recognition and Localization Using Convolutional Neural Network

CNN+RNN Depth and Skeleton based Dynamic Hand Gesture Recognition

Skeleton Based Dynamic Hand Gesture Recognition using LSTM and CNN

MVHANet: Multi-view Hierarchical Aggregation Network for Skeleton-Based Hand Gesture Recognition

Human gesture recognition of dynamic skeleton using graph convolutional networks

A Gesture Recognition Method Based On Spiking Neural Networks For Cognition Development

Sign Language Recognition Based On Facial Expression and Hand Skeleton

Deep self-supervised learning with visualisation for automatic gesture recognition

Skeleton Based Dynamic Hand Gesture Recognition using Short Term Sampling Neural Networks (STSNN).

An Improved Gesture Segmentation Method for Gesture Recognition Based on CNN and YCbCr

Convolutional neural network with spatial pyramid pooling for hand gesture recognition

Visual Hand Gesture Recognition With Convolution Neural Network

Sign language recognition based on skeleton with K3D-LSTMresidual networks

Skeleton Guided Conflict-Free Hand Gesture Recognition for Robot Control.

RGB‐D Static Gesture Recognition Based on Convolutional Neural Network

Skeleton-Based Gesture Recognition Using Several Fully Connected Layers with Path Signature Features and Temporal Transformer Module.

Multi-path Convolutional Neural Network based on Rectangular Kernel with Path Signature Features for Gesture Recognition