Abstract:Vision-based hand gesture interactions are natural and intuitive when interacting with computers, since we naturally exploit gestures to communicate with other people. However, it is agreed that users suffer from discomfort and fatigue when using gesture-controlled interfaces, due to the lack of physical feedback. To solve the problem, we propose a novel complete solution of a hand gesture control system employing immersive tactile feedback to the user's hand. For this goal, we first developed a fast and accurate hand-tracking algorithm with a Kinect sensor using the proposed MLBP (modified local binary pattern) that can efficiently analyze 3D shapes in depth images. The superiority of our tracking method was verified in terms of tracking accuracy and speed by comparing with existing methods, Natural Interaction Technology for End-user (NITE), 3D Hand Tracker and CamShift. As the second step, a new tactile feedback technology with a piezoelectric actuator has been developed and integrated into the developed hand tracking algorithm, including the DTW (dynamic time warping) gesture recognition algorithm for a complete solution of an immersive gesture control system. The quantitative and qualitative evaluations of the integrated system were conducted with human subjects, and the results demonstrate that our gesture control with tactile feedback is a promising technology compared to a vision-based gesture control system that has typically no feedback for the user's gesture inputs. Our study provides researchers and designers with informative guidelines to develop more natural gesture control systems or immersive user interfaces with haptic feedback.

Wearable Drone Controller: Machine Learning-Based Hand Gesture Recognition and Vibrotactile Feedback

Web-based Interactive Drone Control Using Hand Gesture

Advancing Virtual Reality Interaction: A Ring-Shaped Controller and Pose Tracking

Design and Evaluation of an Alternative Control for a Quad-Rotor Drone Using Hand-Gesture Recognition

I Am the UAV: A Wearable Approach for Manipulation of Unmanned Aerial Vehicle

End-Side Gesture Recognition Method for UAV Control

WristCam: A Wearable Sensor for Hand Trajectory Gesture Recognition and Intelligent Human–Robot Interaction

Online Hand Gesture Detection and Recognition for UAV Motion Planning

Virtual Interaction and Manipulation Control of a Hexacopter Through Hand Gesture Recognition from a Data Glove.

Drone Control based on Mental Commands and Facial Expressions

Flyingbuddy2: A Brain-Controlled Assistant For The Handicapped

Personalized Human-Swarm Interaction through Hand Motion

Control of a Drone in Virtual Reality Using MEMS Sensor Technology and Machine Learning

OmniRace: 6D Hand Pose Estimation for Intuitive Guidance of Racing Drone

Dual-hand Gesture Controlled Quadcopter Robot

An Intelligent Human–Unmanned Aerial Vehicle Interaction Approach in Real Time Based on Machine Learning Using Wearable Gloves

DronePick: Object Picking and Delivery Teleoperation with the Drone Controlled by a Wearable Tactile Display

CaptAinGlove: Capacitive and Inertial Fusion-Based Glove for Real-Time on Edge Hand Gesture Recognition for Drone Control

Depth Camera-Based 3D Hand Gesture Controls with Immersive Tactile Feedback for Natural Mid-Air Gesture Interactions

A Wearable and Portable Real-Time Control Gesture Recognition System for Bionic Manipulator

Deep Reinforcement Learning Based Unmanned Aerial Vehicle (UAV) Control Using 3D Hand Gestures