Abstract:Hand gesture and action recognition have been extensively researched in the past two decades due to the emerging advanced acquisition and interaction technologies, which open the floodgates for a vast range of potential applications. Particularly, many spatial–temporal feature extractors have been proposed, such as RNNs-based models, temporal convolutional network (TCN), and 3D convolutional neural networks (3DCNN) for modeling long-term dependencies in sequential data. However, it remains challenging to obtain a high recognition rate because of the difficulty of effectively extracting spatial–temporal features and efficiently classifying them with noisy and complex skeleton sequences. Therefore, this paper proposes a deep ensemble framework called multi-model ensemble gesture recognition network (MMEGRN) for skeleton-based hand gesture recognition. Specifically, to establish effective feature extraction and accurate gesture recognition, we propose an architecture consisting of four sub-networks, three spatio-temporal features classifiers to leverage their various capabilities of extracting and classifying skeleton sequences. Through late feature fusion, the features resulted from the feature extractors of each sub-network are fused into a new fusion classifier. Each subnetwork is trained independently to perform the task of gesture recognition using only skeleton joints. The training is performed using the cyclic annealing learning rate to generate a series of models that are combined in an ensemble using the optimized weighted ensemble (OWE) method. The proposed framework combines deep learning and ensemble strengths to establish a new deep-learning network architecture for more accurate and efficient hand gesture recognition. Extensive experiments on three skeleton-based hand gesture recognition datasets have shown the effectiveness of the proposed framework and the superiority over other models in terms of recognition accuracy.

Recognizing Skeleton-Based Hand Gestures by a Spatio-Temporal Network

Temporal-Spatial Deformable Pose Network for Skeleton-Based Gesture Recognition

A Local Spatial–Temporal Synchronous Network to Dynamic Gesture Recognition

Spatial Temporal Graph Convolutional Networks for Skeleton-Based Dynamic Hand Gesture Recognition

MVHANet: Multi-view Hierarchical Aggregation Network for Skeleton-Based Hand Gesture Recognition

Hidden States Exploration for 3D Skeleton-Based Gesture Recognition.

3D Skeletal Gesture Recognition Via Hidden States Exploration

Skeleton-Based Gesture Recognition Using Several Fully Connected Layers with Path Signature Features and Temporal Transformer Module.

Human gesture recognition of dynamic skeleton using graph convolutional networks

Gesture Recognition Matching Based on Dynamic Skeleton

Graph-Temporal LSTM Networks for Skeleton-Based Action Recognition

Multi-model ensemble gesture recognition network for high-accuracy dynamic hand gesture recognition

Automatic 3D Skeleton-based Dynamic Hand Gesture Recognition Using Multi-Layer Convolutional LSTM.

Spatial-Temporal Attention Res-TCN for Skeleton-Based Dynamic Hand Gesture Recognition

Skeleton-Based Gesture Recognition With Learnable Paths and Signature Features

A Two-stream Neural Network for Pose-based Hand Gesture Recognition

HAN: An Efficient Hierarchical Self-Attention Network for Skeleton-Based Gesture Recognition

Decoupled Representation Learning for Skeleton-Based Gesture Recognition

An Efficient Graph Convolution Network for Skeleton-Based Dynamic Hand Gesture Recognition

Spatio-Temporal Dynamic Attention Graph Convolutional Network Based on Skeleton Gesture Recognition

Spatiotemporal Progressive Inward-Outward Aggregation Network for skeleton-based action recognition