Abstract:Hand gesture and action recognition have been extensively researched in the past two decades due to the emerging advanced acquisition and interaction technologies, which open the floodgates for a vast range of potential applications. Particularly, many spatial–temporal feature extractors have been proposed, such as RNNs-based models, temporal convolutional network (TCN), and 3D convolutional neural networks (3DCNN) for modeling long-term dependencies in sequential data. However, it remains challenging to obtain a high recognition rate because of the difficulty of effectively extracting spatial–temporal features and efficiently classifying them with noisy and complex skeleton sequences. Therefore, this paper proposes a deep ensemble framework called multi-model ensemble gesture recognition network (MMEGRN) for skeleton-based hand gesture recognition. Specifically, to establish effective feature extraction and accurate gesture recognition, we propose an architecture consisting of four sub-networks, three spatio-temporal features classifiers to leverage their various capabilities of extracting and classifying skeleton sequences. Through late feature fusion, the features resulted from the feature extractors of each sub-network are fused into a new fusion classifier. Each subnetwork is trained independently to perform the task of gesture recognition using only skeleton joints. The training is performed using the cyclic annealing learning rate to generate a series of models that are combined in an ensemble using the optimized weighted ensemble (OWE) method. The proposed framework combines deep learning and ensemble strengths to establish a new deep-learning network architecture for more accurate and efficient hand gesture recognition. Extensive experiments on three skeleton-based hand gesture recognition datasets have shown the effectiveness of the proposed framework and the superiority over other models in terms of recognition accuracy.

Temporal-Spatial Deformable Pose Network for Skeleton-Based Gesture Recognition

Recognizing Skeleton-Based Hand Gestures by a Spatio-Temporal Network

Skeleton-Based Gesture Recognition Using Several Fully Connected Layers with Path Signature Features and Temporal Transformer Module.

A Local Spatial–Temporal Synchronous Network to Dynamic Gesture Recognition

Gesture recognition based on deep deformable 3D convolutional neural networks

MVHANet: Multi-view Hierarchical Aggregation Network for Skeleton-Based Hand Gesture Recognition

Spatial Temporal Graph Convolutional Networks for Skeleton-Based Dynamic Hand Gesture Recognition

Motion-Pose Recurrent Neural Network With Instantaneous Kinematic Descriptor For Skeleton Based Gesture Detection And Recognition

Gesture Recognition Matching Based on Dynamic Skeleton

Temporal Decoupling Graph Convolutional Network for Skeleton-Based Gesture Recognition

Multi-path Convolutional Neural Network based on Rectangular Kernel with Path Signature Features for Gesture Recognition

Human gesture recognition of dynamic skeleton using graph convolutional networks

Skeleton-Based Gesture Recognition With Learnable Paths and Signature Features

Automatic 3D Skeleton-based Dynamic Hand Gesture Recognition Using Multi-Layer Convolutional LSTM.

Hidden States Exploration for 3D Skeleton-Based Gesture Recognition.

Multi-model ensemble gesture recognition network for high-accuracy dynamic hand gesture recognition

Spatio-Temporal Dynamic Attention Graph Convolutional Network Based on Skeleton Gesture Recognition

3D Skeletal Gesture Recognition Via Hidden States Exploration

Spatial-Temporal Attention Res-TCN for Skeleton-Based Dynamic Hand Gesture Recognition

Dynamic Gesture Recognition Based on Feature Fusion Network and Variant ConvLSTM.

Joints-Centered Spatial-Temporal Features Fused Skeleton Convolution Network for Action Recognition