Abstract:Abstract. In this era, intelligent vision computing has always been a fascinating field. With the rapid development in computer vision, dynamic gesture-based recognition systems have attracted significant attention. However, automatically recognizing skeleton-based human gestures in the form of sign language is complex and challenging. Most existing methods consider skeleton-based human gesture recognition as a standard video recognition problem, without considering the rich structure information among both joints and gesture frames. Graph convolutional networks (GCNs) are a promising way to leverage structure information to learn structure representations. However, adopting GCNs to tackle such gesture sequences both in spatial and temporal spaces is challenging as graph could be highly nonlinear and complex. To overcome this issue, we propose the spatiotemporal GCNs model to leverage the powerful spatiotemporal correlations to adaptively construct spatiotemporal graphs, called Aegles. Our method could dynamically attend to relatively significant spatiotemporal joints and construct different graphs, including spatial, temporal, and spatiotemporal graph, and well capturing the structure information in gesture sequences. Besides, we introduce the second-order information of the gesture skeleton data, i.e., the length and orientation of bones, to improve the representation of human hands and fingers. In addition, with the public sign language datasets, we use OpenPose technology to extract human gesture skeleton and obtain human skeleton video, building four skeleton-based sign language recognition datasets. Experimental results show that this Aegles outperforms the state-of-the-art ones and that the spatiotemporal correlations effectively boost the performance of human gesture recognition.

Structured Streaming Skeleton -- A New Feature for Online Human Gesture Recognition

Online human gesture recognition from motion data streams.

Skeleton-Based Gesture Recognition Using Several Fully Connected Layers with Path Signature Features and Temporal Transformer Module.

Online Robust Action Recognition Based on a Hierarchical Model

Online Human Action Recognition with Spatial and Temporal Skeleton Features Using a Distributed Camera Network.

3D Articulated Skeleton Extraction Using a Single Consumer-Grade Depth Camera.

Recognizing Skeleton-Based Hand Gestures by a Spatio-Temporal Network

Online Kernel-Based Structured Output SVM for Early Expression Detection

A Local Spatial–Temporal Synchronous Network to Dynamic Gesture Recognition

Hidden States Exploration for 3D Skeleton-Based Gesture Recognition.

Real-Time Hand Gesture Recognition: Integrating Skeleton-Based Data Fusion and Multi-Stream CNN

Online Real-Time Learning Strategies for Data Streams for Neurocomputing

Human gesture recognition of dynamic skeleton using graph convolutional networks

Real-Time Online Skeleton Extraction and Gesture Recognition on Pepper

Vision-Based Gesture Recognition Referring to Human Structure

Skeleton Based Dynamic Hand Gesture Recognition using Short Term Sampling Neural Networks (STSNN).

Temporal-Spatial Deformable Pose Network for Skeleton-Based Gesture Recognition

Online gesture spotting from visual hull data.

Automatic 3D Skeleton-based Dynamic Hand Gesture Recognition Using Multi-Layer Convolutional LSTM.

Skeleton-based Dynamic Hand Gesture Recognition using 3D Depth Data

Skeleton Feature Fusion Based on Multi-Stream LSTM for Action Recognition.