Abstract:In this work, a real-time hand gesture recognition system-based human-computer interface (HCI) is presented. The system consists of six stages: (1) hand detection, (2) gesture segmentation, (3) use of five pre-trained convolutional neural network models (CNN) and vision transformer (ViT), (4) building an interactive human-machine interface (HMI), (5) development of a gesture-controlled virtual mouse, (6) use of Kalman filter to estimate the hand position, based on that the smoothness of the motion of pointer is improved. In our work, five pre-trained CNN (VGG16, VGG19, ResNet50, ResNet101, and Inception-V1) models and ViT have been employed to classify hand gesture images. Two multi-class datasets (one public and one custom) have been used to validate the models. Considering the model's performances, it is observed that Inception-V1 has significantly shown a better classification performance compared to the other four CNN models and ViT in terms of accuracy, precision, recall, and F-score values. We have also expanded this system to control some desktop applications (such as VLC player, audio player, file management, playing 2D Super-Mario-Bros game, etc.) with different customized gesture commands in real-time scenarios. The average speed of this system has reached 25 fps (frames per second), which meets the requirements for the real-time scenario. Performance of the proposed gesture control system obtained the average response time in milisecond for each control which makes it suitable for real-time. This model (prototype) will benefit physically disabled people interacting with desktops.

Real time robotic arm control using hand gestures

Real-Time Robotic Hand Control Using Hand Gestures

Vision-Based Real-Time Human–Computer Interaction on Hand Gesture Recognition

Real Time Robotic Arm using Leap Motion Controller

WristCam: A Wearable Sensor for Hand Trajectory Gesture Recognition and Intelligent Human–Robot Interaction

A Prototype System for High Frame Rate Ultrasound Imaging based Prosthetic Arm Control

Hand Gesture Based Robot Control System Using Leap Motion

Hand-Eye Servo And Flexible Control Of An Anthropomorphic Arm

of Advanced Robotic Systems Hand Motion-Based Remote Control Interface with Vibrotactile Feedback for Home Robots Regular Paper

Exploring The Techniques and Applications of Hand Gesture Recognition for Human-Computer Interaction

Visual Control of a Robot Head.

Real-time Vision-based Gesture Recognition for Human Robot Interaction

Hand Motion-Based Remote Control Interface With Vibrotactile Feedback For Home Robots

A Real Time Vision-Based Hand Gestures Recognition System

The Manipulation of Real-Time Kinect-Based Robotic Arm Using Double-Hand Gestures

Visual Control of a Robot Head

An IMUs and potentiometer-based controller for robotic arm-hand teleoperation

Real-Time Human-Computer Interaction Based on Face and Hand Gesture Recognition

Deep learning based Hand gesture recognition system and design of a Human-Machine Interface

Hand gesture recognition and voice controlled robot