Abstract:In this work, a real-time hand gesture recognition system-based human-computer interface (HCI) is presented. The system consists of six stages: (1) hand detection, (2) gesture segmentation, (3) use of five pre-trained convolutional neural network models (CNN) and vision transformer (ViT), (4) building an interactive human-machine interface (HMI), (5) development of a gesture-controlled virtual mouse, (6) use of Kalman filter to estimate the hand position, based on that the smoothness of the motion of pointer is improved. In our work, five pre-trained CNN (VGG16, VGG19, ResNet50, ResNet101, and Inception-V1) models and ViT have been employed to classify hand gesture images. Two multi-class datasets (one public and one custom) have been used to validate the models. Considering the model's performances, it is observed that Inception-V1 has significantly shown a better classification performance compared to the other four CNN models and ViT in terms of accuracy, precision, recall, and F-score values. We have also expanded this system to control some desktop applications (such as VLC player, audio player, file management, playing 2D Super-Mario-Bros game, etc.) with different customized gesture commands in real-time scenarios. The average speed of this system has reached 25 fps (frames per second), which meets the requirements for the real-time scenario. Performance of the proposed gesture control system obtained the average response time in milisecond for each control which makes it suitable for real-time. This model (prototype) will benefit physically disabled people interacting with desktops.

Hand gesture recognition and voice controlled robot

Gesture and Voice Controlled Robot for Industrial Applications

Dynamic Hand Gesture-Featured Human Motor Adaptation in Tool Delivery using Voice Recognition

AI Based Virtual Mouse with Hand Gesture and AI Voice Assistant Using Computer Vision and Neural Networks

Vision-Based Real-Time Human–Computer Interaction on Hand Gesture Recognition

Hand Gesture Control for Human–Computer Interaction with Deep Learning

Exploring The Techniques and Applications of Hand Gesture Recognition for Human-Computer Interaction

Deep learning based Hand gesture recognition system and design of a Human-Machine Interface

Gesture recognition using combination of acceleration sensor and images for casual communication between robots and humans

Real time robotic arm control using hand gestures

Low-Cost Robotic Arm for differently abled using Voice Recognition

Interactive Control of a Robotic Arm using Voice Recognition

Gesture Controlled Virtual Mouse with Voice Assistant

A Hand Gestures Recognition System for Industrial Robot Control

Gesture Recognition Based on Localist Attractor Networks with Application to Robot Control [Application Notes]

A Novel Gesture Recognition System for Intelligent Interaction with a Nursing-Care Assistant Robot

Real-time Vision-based Gesture Recognition for Human Robot Interaction

Hands-On Robotics: Enabling Communication Through Direct Gesture Control

Gesture Recognition for Human-Robot Interaction Through a Knowledge Based Software Platform

Development of a Real Time Vision-Based Hand Gesture Recognition System for Human-Computer Interaction

Efficient Hand Gesture Recognition for Human-Robot Interaction