Abstract:Robust segmentation of hands in a cluttered environment for hand gesture recognition has remained a challenge in computer vision. In this work, a two-stage gesture recognition framework is proposed. In the first stage, we segment hands using the proposed deep learning algorithm, and in the second stage, we use these segmented hands to classify gestures using a novel structurally incoherent nonnegative matrix factorization approach. We propose a new deep learning framework for hand segmentation called densely dilated U-Net. We exploit recently proposed dense blocks and dilated convolution layers in our work. To cope with the scarcity of labeled datasets we extend our densely dilated U-Net for semisupervised hand segmentation using hand bounding boxes as cues. We provide quantitative and qualitative evaluation of proposed hand segmentation model on several public hand segmentation datasets including EgoHands, GTEA, EYTH, EDSH, and HOF. Semisupervised segmentation results are also obtained on two hand detection datasets including VIVA and CVRR. As an extension of our work, we show semisupervised segmentation and gesture recognition results using segmented hands on NUS-II cluttered hand gesture dataset. To validate the efficiency of our semisupervised algorithm we evaluate it on OUHands dataset with real ground truth labels. For gesture classification, we propose a novel structurally incoherent nonnegative matrix factorization algorithm. We propose to use CNN features extracted from segmented images for nonnegative matrix factorization. Experimental results on NUS-II and OUHands datasets demonstrate that our two-stage approach for gesture recognition yields superior results.

Hand Segmentation and Fingertip Tracking from Depth Camera Images Using Deep Convolutional Neural Network and Multi-task SegNet

Real-Time Hand Gesture Spotting and Recognition Using RGB-D Camera and 3D Convolutional Neural Network

Precise Hand Segmentation from a Single Depth Image

A New Hand Segmentation Method Based on Fully Convolutional Network

Hand Segmentation for Hand-Object Interaction from Depth map

SegPoseNet: Segmentation-Guided 3D Hand Pose Estimation

Hand Segmentation With Dense Dilated U-Net and Structurally Incoherent Nonnegative Matrix Factorization-Based Gesture Recognition

Hand segmentation pipeline from depth map: an integrated approach of histogram threshold selection and shallow CNN classification

Two-stream Convolutional Neural Network for Accurate RGB-D Fingertip Detection Using Depth and Edge Information.

Real-time Fingertip Tracking and Gesture Recognition Using RGB-D Camera

Real-Time Hand Gesture Recognition Using Finger Segmentation

Deep Attention Network for Joint Hand Gesture Localization and Recognition Using Static RGB-D Images.

Fingertip in the Eye: An Attention-Based Method for Real-Time Hand Tracking and Fingertip Detection in Egocentric Videos

Continuous Gesture Segmentation and Recognition Using 3DCNN and Convolutional LSTM

Attention Based Dual Branches Fingertip Detection Network and Virtual Key System

Deep Multi-Task Learning based Fingertip Detection

Real-time Fingertip Detection Based on Depth Data

Hand Gesture Recognition with Skin Detection and Deep Learning Method

Deep Network Based 3D Hand Keypoints Prediction from Single RGB Images

Research on the Hand Gesture Recognition Based on Deep Learning

Point Cloud Hand–Object Segmentation Using Multimodal Imaging with Thermal and Color Data for Safe Robotic Object Handover