Abstract:Purpose The purpose of this paper is to mainly center on the touchless interaction between humans and robots in the real world. The accuracy of hand pose identification and stable operation in a non-stationary environment is the main challenge, especially in multiple sensors conditions. To guarantee the human-machine interaction system’s performance with a high recognition rate and lower computational time, an adaptive sensor fusion labeling framework should be considered in surgery robot teleoperation. Design/methodology/approach In this paper, a hand pose estimation model is proposed consisting of automatic labeling and classified based on a deep convolutional neural networks (DCNN) structure. Subsequently, an adaptive sensor fusion methodology is proposed for hand pose estimation with two leap motions. The sensor fusion system is implemented to process depth data and electromyography signals capturing from Myo Armband and leap motion, respectively. The developed adaptive methodology can perform stable and continuous hand position estimation even when a single sensor is unable to detect a hand. Findings The proposed adaptive sensor fusion method is verified with various experiments in six degrees of freedom in space. The results showed that the clustering model acquires the highest clustering accuracy (96.31%) than other methods, which can be regarded as real gestures. Moreover, the DCNN classifier gets the highest performance (88.47% accuracy and lowest computational time) than other methods. Originality/value This study can provide theoretical and engineering guidance for hand pose recognition in surgery robot teleoperation and design a new deep learning model for accuracy enhancement.

BaSICNet: Lightweight 3D Hand Pose Estimation Network Based on Biomechanical Structure Information for Dexterous Manipulator Teleoperation

Dual-Hand Motion Capture by Using Biological Inspiration for Bionic Bimanual Robot Teleoperation

A Dexterous Hand-Arm Teleoperation System Based on Hand Pose Estimation and Active Vision

Applying 3D Human Hand Pose Estimation to Teleoperation

Weakly Supervised 3D Hand Pose Estimation via Biomechanical Constraints

A 3D Hand Attitude Estimation Method for Fixed Hand Posture Based on Dual-View RGB Images

3D Capsule Hand Pose Estimation Network Based on Structural Relationship Information

Adaptive sensor fusion labeling framework for hand pose recognition in robot teleoperation

BiHand: Recovering Hand Mesh with Multi-stage Bisected Hourglass Networks

Hand3D: Hand Pose Estimation using 3D Neural Network

Recurrent 3D Hand Pose Estimation Using Cascaded Pose-Guided 3D Alignments

Improved Deep Learning-Based Efficientpose Algorithm for Egocentric Marker-Less Tool and Hand Pose Estimation in Manual Assembly.

Mmhand: 3D Hand Pose Estimation Leveraging Mmwave Signals

Estimate Hand Poses Efficiently from Single Depth Images

Biomimetic Evaluation of an Underwater Soft Hand Through Deep Learning-based 3D Pose Reconstruction

Accurate 3D Hand Pose Estimation Network Utilizing Joints Information.

Hand Pose Estimation with Attention-and-Sequence Network.

DeepHPS: End-to-end Estimation of 3D Hand Pose and Shape by Learning from Synthetic Depth

A hybrid network for estimating 3D interacting hand pose from a single RGB image

Toward Marker-free 3D Pose Estimation in Lifting: A Deep Multi-view Solution

Multistage 3D Hand Pose Estimation Algorithm Based on Skeleton Points