Abstract:Online hand gesture recognition (HGR) techniques are essential in augmented reality (AR) applications for enabling natural human-to-computer interaction and communication. In recent years, the consumer market for low-cost AR devices has been rapidly growing, while the technology maturity in this domain is still limited. Those devices are typical of low prices, limited memory, and resource-constrained computational units, which makes online HGR a challenging problem. To tackle this problem, we propose a lightweight and computationally efficient HGR framework, namely LE-HGR, to enable real-time gesture recognition on embedded devices with low computing power. We also show that the proposed method is of high accuracy and robustness, which is able to reach high-end performance in a variety of complicated interaction environments. To achieve our goal, we first propose a cascaded multi-task convolutional neural network (CNN) to simultaneously predict probabilities of hand detection and regress hand keypoint locations online. We show that, with the proposed cascaded architecture design, false-positive estimates can be largely eliminated. Additionally, an associated mapping approach is introduced to track the hand trace via the predicted locations, which addresses the interference of multi-handedness. Subsequently, we propose a trace sequence neural network (TraceSeqNN) to recognize the hand gesture by exploiting the motion features of the tracked trace. Finally, we provide a variety of experimental results to show that the proposed framework is able to achieve state-of-the-art accuracy with significantly reduced computational cost, which are the key properties for enabling real-time applications in low-cost commercial devices such as mobile devices and AR/VR headsets.

A Two-Branch Hand Gesture Recognition Approach Combining Atrous Convolution and Attention Mechanism

MVHANet: Multi-view Hierarchical Aggregation Network for Skeleton-Based Hand Gesture Recognition

Mask-guided Multiscale Feature Aggregation Network for Hand Gesture Recognition

Multimodal Gesture Recognition Using 3-D Convolution and Convolutional LSTM

A Robust Hand Gesture Sensing and Recognition Based on Dual-Flow Fusion with FMCW Radar.

Multi-Scale Attention 3D Convolutional Network for Multimodal Gesture Recognition

A Multimodal Dynamic Hand Gesture Recognition Based on Radar–Vision Fusion

Continuous Gesture Segmentation and Recognition Using 3DCNN and Convolutional LSTM

A Hybrid Multimodal Fusion Framework for Semg-Acc-based Hand Gesture Recognition

Large-scale Multimodal Gesture Segmentation and Recognition Based on Convolutional Neural Networks

A Multi-stream Convolutional Model Using Attention Module for Semg Gesture Recognition

Hand Segmentation With Dense Dilated U-Net and Structurally Incoherent Nonnegative Matrix Factorization-Based Gesture Recognition

Multi-Task and Multi-Modal Learning for RGB Dynamic Gesture Recognition

A Robust Hand Gesture Recognition Method Via Convolutional Neural Network

A Two-stream Neural Network for Pose-based Hand Gesture Recognition

Gesture Recognition Method Based on Attention Mechanism for Complex Background

Spatial-temporal Dynamic Hand Gesture Recognition Via Hybrid Deep Learning Model

LE-HGR: A Lightweight and Efficient RGB-based Online Gesture Recognition Network for Embedded AR Devices

Hand Gesture Recognition Based on Attentive Feature Fusion

2D Hand Detection Using Multi-Feature Skin Model Supervised Cascaded CNN

FMCW Radar-Based Hand Gesture Recognition Using Spatiotemporal Deformable and Context-Aware Convolutional 5-D Feature Representation