Abstract:Purpose The purpose of this paper is to mainly center on the touchless interaction between humans and robots in the real world. The accuracy of hand pose identification and stable operation in a non-stationary environment is the main challenge, especially in multiple sensors conditions. To guarantee the human-machine interaction system’s performance with a high recognition rate and lower computational time, an adaptive sensor fusion labeling framework should be considered in surgery robot teleoperation. Design/methodology/approach In this paper, a hand pose estimation model is proposed consisting of automatic labeling and classified based on a deep convolutional neural networks (DCNN) structure. Subsequently, an adaptive sensor fusion methodology is proposed for hand pose estimation with two leap motions. The sensor fusion system is implemented to process depth data and electromyography signals capturing from Myo Armband and leap motion, respectively. The developed adaptive methodology can perform stable and continuous hand position estimation even when a single sensor is unable to detect a hand. Findings The proposed adaptive sensor fusion method is verified with various experiments in six degrees of freedom in space. The results showed that the clustering model acquires the highest clustering accuracy (96.31%) than other methods, which can be regarded as real gestures. Moreover, the DCNN classifier gets the highest performance (88.47% accuracy and lowest computational time) than other methods. Originality/value This study can provide theoretical and engineering guidance for hand pose recognition in surgery robot teleoperation and design a new deep learning model for accuracy enhancement.

A Novel Sensor Fusion Approach for Precise Hand Tracking in Virtual Reality-Based Human—Computer Interaction

Advancing Virtual Reality Interaction: A Ring-Shaped Controller and Pose Tracking

2D Motion Detection Bounded Hand 3D Trajectory Tracking and Gesture Recognition under Complex Background

A Lightweight and Efficient System for Tracking Handheld Objects in Virtual Reality

Interaction and control with the auxiliary of hand gesture

Controller-free Hand Tracking for Grab-and-place Tasks in Immersive Virtual Reality: Design Elements and their Empirical Study

Toward Intuitive 3D Interactions in Virtual Reality: A Deep Learning- Based Dual-Hand Gesture Recognition Approach

Novel Adaptive Sensor Fusion Methodology for Hand Pose Estimation with Multileap Motion

WristCam: A Wearable Sensor for Hand Trajectory Gesture Recognition and Intelligent Human–Robot Interaction

High-Fidelity Grasping In Virtual Reality Using A Glove-Based System

Multi-modal Sensing-Based Interactive Glove System for Teleoperation and VR/AR

A touchable virtual screen interaction system with handheld Kinect camera

Virtual-Physical Interaction System

Arm Motion Estimation Based on Predictive Control

Hand Gesture Recognition using Deep Feature Fusion Network based on Wearable Sensors

A Motion-based User Interface for the Control of Virtual Humans Performing Sports.

Adaptive sensor fusion labeling framework for hand pose recognition in robot teleoperation

Visual-inertial hand motion tracking with robustness against occlusion, interference, and contact

Hand Tracking: Survey

Real-Time Hand Gesture Tracking for Human–Computer Interface Based on Multi-Sensor Data Fusion

V-Hands: Touchscreen-based Hand Tracking for Remote Whiteboard Interaction