Abstract:Hand gesture and action recognition have been extensively researched in the past two decades due to the emerging advanced acquisition and interaction technologies, which open the floodgates for a vast range of potential applications. Particularly, many spatial–temporal feature extractors have been proposed, such as RNNs-based models, temporal convolutional network (TCN), and 3D convolutional neural networks (3DCNN) for modeling long-term dependencies in sequential data. However, it remains challenging to obtain a high recognition rate because of the difficulty of effectively extracting spatial–temporal features and efficiently classifying them with noisy and complex skeleton sequences. Therefore, this paper proposes a deep ensemble framework called multi-model ensemble gesture recognition network (MMEGRN) for skeleton-based hand gesture recognition. Specifically, to establish effective feature extraction and accurate gesture recognition, we propose an architecture consisting of four sub-networks, three spatio-temporal features classifiers to leverage their various capabilities of extracting and classifying skeleton sequences. Through late feature fusion, the features resulted from the feature extractors of each sub-network are fused into a new fusion classifier. Each subnetwork is trained independently to perform the task of gesture recognition using only skeleton joints. The training is performed using the cyclic annealing learning rate to generate a series of models that are combined in an ensemble using the optimized weighted ensemble (OWE) method. The proposed framework combines deep learning and ensemble strengths to establish a new deep-learning network architecture for more accurate and efficient hand gesture recognition. Extensive experiments on three skeleton-based hand gesture recognition datasets have shown the effectiveness of the proposed framework and the superiority over other models in terms of recognition accuracy.

Skeleton-Based Dynamic Hand Gesture Recognition Using an Enhanced Network with One-Shot Learning

MVHANet: Multi-view Hierarchical Aggregation Network for Skeleton-Based Hand Gesture Recognition

Skeleton Based Dynamic Hand Gesture Recognition using Short Term Sampling Neural Networks (STSNN).

Motion Feature Augmented Recurrent Neural Network for Skeleton-based Dynamic Hand Gesture Recognition

MFA-Net: Motion Feature Augmented Network for Dynamic Hand Gesture Recognition from Skeletal Data

Skeleton-Based Gesture Recognition Using Several Fully Connected Layers with Path Signature Features and Temporal Transformer Module.

DHGD: Dynamic Hand Gesture Dataset for Skeleton-Based Gesture Recognition and Baseline Evaluations

Multi-model ensemble gesture recognition network for high-accuracy dynamic hand gesture recognition

A Local Spatial–Temporal Synchronous Network to Dynamic Gesture Recognition

Gesture Recognition Matching Based on Dynamic Skeleton

An Efficient Graph Convolution Network for Skeleton-Based Dynamic Hand Gesture Recognition

Skeleton Based Dynamic Hand Gesture Recognition using LSTM and CNN

CNN+RNN Depth and Skeleton based Dynamic Hand Gesture Recognition

Temporal-Spatial Deformable Pose Network for Skeleton-Based Gesture Recognition

Dynamic Hand Gesture Recognition Based On 3D Skeleton

A Skeleton Object Detection-Based Dynamic Gesture Recognition Method

Spatio-Temporal Dynamic Attention Graph Convolutional Network Based on Skeleton Gesture Recognition

Adaptive Local Spatiotemporal Features from RGB-D Data for One-Shot Learning Gesture Recognition.

Skeleton-based Dynamic Hand Gesture Recognition using 3D Depth Data

HAN: An Efficient Hierarchical Self-Attention Network for Skeleton-Based Gesture Recognition

Spatial Temporal Graph Convolutional Networks for Skeleton-Based Dynamic Hand Gesture Recognition