Abstract:This paper presents an integrated mapping of motion and visualization scheme based on a Mixed Reality (MR) subspace approach for the intuitive and immersive telemanipulation of robotic arm-hand systems. The effectiveness of different control-feedback methods for the teleoperation system is validated and compared. The robotic arm-hand system consists of a 6 Degrees-of-Freedom (DOF) industrial manipulator and a low-cost 2-finger gripper, which can be manipulated in a natural manner by novice users physically distant from the working site. By incorporating MR technology, the user is fully immersed in a virtual operating space augmented by real-time 3D visual feedback from the robot working site. Imitation-based velocity-centric motion mapping is implemented via the MR subspace to accurately track operator hand movements for robot motion control and enables spatial velocity-based control of the robot Tool Center Point (TCP). The user control space and robot working space are overlaid through the MR subspace, and the local user and a digital twin of the remote robot share the same environment in the MR subspace. The MR-based motion and visualization mapping scheme for telerobotics is compared to conventional 2D Baseline and MR tele-control paradigms over two tabletop object manipulation experiments. A user survey of 24 participants was conducted to demonstrate the effectiveness and performance enhancements enabled by the proposed system. The MR-subspace-integrated 3D mapping of motion and visualization scheme reduced the aggregate task completion time by 48% compared to the 2D Baseline module and 29%, compared to the MR SpaceMouse module. The perceived workload decreased by 32% and 22%, compared to the 2D Baseline and MR SpaceMouse approaches.

Human-machine Interface for a Master-Slave Hydraulic Manipulator with Vision Enhancement and Auditory Feedback

A Teleoperation Framework Based on Heterogeneous Matching for Hydraulic Manipulator

Multitarget Adaptive Virtual Fixture Based on Task Learning for Hydraulic Manipulator

A Multi‐Module Sensing and Bi‐Directional HMI Integrating Interaction, Recognition, and Feedback for Intelligent Robots

A Wearable Human–Machine Interactive Instrument for Controlling a Wheelchair Robotic Arm System

Development of human-machine interface for teleoperation of a mobile manipulator

Position-Based Visual Servoing Control for Multi-Joint Hydraulic Manipulator

Human–Robot Shared Control Based on Locally Weighted Intent Prediction for a Teleoperated Hydraulic Manipulator System

Mechanical Acoustic Signal Assisted Translational Model for Industrial Human-Machine Interaction

Evaluation of Human-Robot Interfaces based on 2D/3D Visual and Haptic Feedback for Aerial Manipulation

Human-Manipulator Interface Using Hybrid Sensors Via CMAC for Dual Robots.

Evaluating visual encoding quality of a mixed reality user interface for human–machine co-assembly in complex operational terrain

Human-Machine Interface for Remote Training of Robot Tasks

Mixed reality-integrated 3D/2D vision mapping for intuitive teleoperation of mobile manipulator

A Wearable-Based and Markerless Human-Manipulator Interface with Feedback Mechanism and Kalman Filters

Design and Evaluation of an Augmented Reality Head-Mounted Display Interface for Human Robot Teams Collaborating in Physically Shared Manufacturing Tasks

Toward a Human-Machine Interface Based on Electrical Impedance Tomography for Robotic Manipulator Control.

Mixed-Reality-Enhanced Human–Robot Interaction with an Imitation-Based Mapping Approach for Intuitive Teleoperation of a Robotic Arm-Hand System

Human-Machine Engineering Virtual Simulation for the Wheelhouse on Large Ships

Development and Application of Integrated Human Machine Interaction Simulation System

Persistent Human–Machine Interfaces for Robotic Arm Control Via Gaze and Eye Direction Tracking