Abstract:Obtaining robust fine-grained behavioral features is critical for dynamic hand gesture authentication. However, behavioral characteristics are abstract and complex, making them more difficult to capture than physiological characteristics. Moreover, various illumination and backgrounds in practical applications pose additional challenges to existing methods because commonly used RGB videos are sensitive to them. To overcome this robustness limitation, we propose a two-stream CNN-based cross-modality local-global network (CMLG-Net) with two complementary modules to enhance the discriminability and robustness of behavioral features. First, we introduce a temporal scale pyramid (TSP) module consisting of multiple parallel convolution subbranches with different temporal kernel sizes to capture the fine-grained local motion cues at various temporal scales. Second, a cross-modality temporal non-local (CMTNL) module is devised to simultaneously aggregate the global temporal features and cross-modality features with an attention mechanism. Through the complementary combination of the TSP and CMTNL modules, our CMLG-Net obtains a comprehensive and robust behavioral representation that contains both multi-scale (short- and long-term) and multimodal (RGB-D) behavioral information. Extensive experiments are conducted on the largest dataset, SCUT-DHGA, and a simulated practical dataset, SCUT-DHGA-br, to demonstrate the effectiveness of CMLG-Net in exploiting fine-grained behavioral features and complementary multimodal information. Finally, it achieves stat-of-the-art performance with the lowest ERR of 0.497% and 4.848% in two challenging evaluation protocols and shows significant superiority in robustness under practical scenes with unsatisfactory illumination and backgrounds. The code is available at https://github.com/SCUT-BIP-Lab/CMLG-Net.

An Improved Gesture Segmentation Method for Gesture Recognition Based on CNN and YCbCr

A real-time hand gesture recognition method

Visual Hand Gesture Recognition With Convolution Neural Network

Continuous Gesture Segmentation and Recognition Using 3DCNN and Convolutional LSTM

Gesture Feature Extraction and Recognition Based on Image Processing

Hand Gesture Recognition Based Improved Multi-Channels CNN Architecture Using EMG Sensors

Large-scale Multimodal Gesture Segmentation and Recognition Based on Convolutional Neural Networks

A Recognition Method of Hand Gesture with CNN-SVM Model.

Gesture Recognition Based on Skeletonization Algorithm and CNN with ASL Database

Gesture Recognition and Localization Using Convolutional Neural Network

Hand Gesture Recognition in Complex Background Based on Improved Deep Residual Learning Network

An Improved Algorithm of Hand Gesture Recognition under Intricate Background

Gesture Recognition Algorithm Based on Multi-Scale Feature Fusion in RGB-D Images

G3-2d: A Continuous Gesture Segmentation and Recognition Model to Shoot Gestures from 3d Space to 2d Coordinate System

Robust and Accurate Hand Gesture Authentication with Cross-Modality Local-Global Behavior Analysis

A Video Gesture Processing Method Based on Convolution and Long Short-Term Memory Network

Multimodal Gesture Recognition Based on the ResC3D Network.

A Robust Hand Gesture Recognition Method Via Convolutional Neural Network

Static Gesture Segmentation Technique Based on Improved Sobel Operator

Optimizing Gesture Recognition for Seamless UI Interaction Using Convolutional Neural Networks

An Improved Otsu's Thresholding Algorithm on Gesture Segmentation