Abstract:Gesture recognition is getting more and more popular due to various application possibilities in human-machine interaction. Existing multi-modal gesture recognition systems take multi-modal data as input to improve accuracy, but such methods require more modality sensors, which will greatly limit their application scenarios. Therefore we propose an end-to-end multi-task learning framework in training 2D convolutional neural networks. The framework can use the depth modality to improve accuracy during training and save costs by using only RGB modality during inference. Our framework is trained to learn a representation for multi-task learning: gesture segmentation and gesture recognition. Depth modality contains the prior information for the location of the gesture. Therefore it can be used as the supervision for gesture segmentation. A plug-and-play module named Multi-Scale-Decoder is designed to realize gesture segmentation, which contains two sub-decoder. It is used in the lower stage and higher stage respectively, and can help the network pay attention to key target areas, ignore irrelevant information, and extract more discriminant features. Additionally, the MSD module and depth modality are only used in the training stage to improve gesture recognition performance. Only RGB modality and network without MSD are required during inference. Experimental results on three public gesture recognition datasets show that our proposed method provides superior performance compared with existing gesture recognition frameworks. Moreover, using the proposed plug-and-play MSD in other 2D CNN-based frameworks also get an excellent accuracy improvement.

Multi-path Convolutional Neural Network based on Rectangular Kernel with Path Signature Features for Gesture Recognition

Skeleton-Based Gesture Recognition With Learnable Paths and Signature Features

Skeleton-Based Gesture Recognition Using Several Fully Connected Layers with Path Signature Features and Temporal Transformer Module.

Multimodal Gesture Recognition Using 3-D Convolution and Convolutional LSTM

Temporal-Spatial Deformable Pose Network for Skeleton-Based Gesture Recognition

MVHANet: Multi-view Hierarchical Aggregation Network for Skeleton-Based Hand Gesture Recognition

Multi-Scale Attention 3D Convolutional Network for Multimodal Gesture Recognition

Modality-convolutions: Multi-modal gesture recognition based on convolutional neural network

Gesture Recognition and Localization Using Convolutional Neural Network

Spatial Temporal Graph Convolutional Networks for Skeleton-Based Dynamic Hand Gesture Recognition

Multimode Gesture Recognition Algorithm Based on Convolutional Long Short-Term Memory Network

Large-scale Isolated Gesture Recognition Using Convolutional Neural Networks

Large-Scale Multimodal Gesture Recognition Using Heterogeneous Networks

Gesture Recognition Based on Skeletonization Algorithm and CNN with ASL Database

LPSNet: A Novel Log Path Signature Feature Based Hand Gesture Recognition Framework.

Decoupled Representation Learning for Skeleton-Based Gesture Recognition

Large-scale Multimodal Gesture Segmentation and Recognition Based on Convolutional Neural Networks

An Efficient Graph Convolution Network for Skeleton-Based Dynamic Hand Gesture Recognition

Multi-Task and Multi-Modal Learning for RGB Dynamic Gesture Recognition

Multi-layered Gesture Recognition with Kinect

Automatic 3D Skeleton-based Dynamic Hand Gesture Recognition Using Multi-Layer Convolutional LSTM.