Abstract:In recent years Virtual Reality (VR) and Augmented Reality (AR) Head-Mounted Displays (HMDs) have been used to provide an immersive, first-person view in real-time in robotic teleoperation applications.However, there are two main critical issues in robotic teleoperation for remote manipulation and real-world interaction: firstly, it is challenging to correctly perceive the distance and the dimensions during fine manipulation tasks; secondly, latency in video streaming can affect the immersive experience of the operator. To address these issues, in this paper we propose a novel system that allows the operator to immersively teleoperate robotic arms using his/her hand motion and gesture.More in detail, the operator wears a HMD equipped with hand tracking and head orientation estimation technologies. The hand tracking and gesture recognition are used by the operator to generate control commands for remote robotic arms, whereas the head orientation angles are used to control the orientation of a remote robotic base on which is installed a RGB-Depth (RGB-D) camera. The latter allows the operator an improved immersive experience since he/she can freely rotate his/her head resulting in a corresponding movement of the camera on the remote side. Furthermore, the video/depth streaming of the RGB-D camera is used for three main objectives: i) appropriately filtering the RGB image through spatial depth information for reducing the number of significant pixels to be sent to the HMD thus reducing the latency; ii) providing spatial cues overlaid on the 2D image to facilitate remote interaction (basic AR); iii) generating additional augmented information with the real-world environment for specific tasks (advanced AR). On the remote side of the system, the robotic base of the RGB-D camera is integral with the base system of two robotic arms, each of them equipped with a robotic hand for object manipulation and interaction.The proposed system can be used for several tasks in several scenarios including social applications and high-risk situations. In the former scenario, an operator can remotely interact with a person by providing assistance or by playing board games. In the latter scenario, our system, suitably integrated with a robotic base allowing its mobility, makes possible some teleoperated activities such as door opening, valve turning and switch operating without local human intervention.

Semi-autonomous Robotic Disassembly Enhanced by Mixed Reality

Electric Vehicle Battery Disassembly Using Interfacing Toolbox for Robotic Arms

Interactive control of manufacturing assemblies with Mixed Reality

Towards Reuse and Recycling of Lithium-ion Batteries: Tele-robotics for Disassembly of Electric Vehicle Batteries

Robotic Disassembly and Remanufacturing Automation.

A Mixed-Reality Tele-Operation Method for High-Level Control of a Legged-Manipulator Robot

Design of an Energy-Aware Cartesian Impedance Controller for Collaborative Disassembly

Closed-Loop Robotic Arm Manipulation Based on Mixed Reality

An Experimental Human-Robot Collaborative Disassembly Cell

On the Fly Robotic-Assisted Medical Instrument Planning and Execution Using Mixed Reality

IMMERTWIN: A Mixed Reality Framework for Enhanced Robotic Arm Teleoperation

An Augmented Reality-Assisted Disassembly Approach for End-of-Life Vehicle Power Batteries

An augmented-reality-assisted immersive system for robotic arms teleoperation

Revolutionizing Battery Disassembly: The Design and Implementation of a Battery Disassembly Autonomous Mobile Manipulator Robot(BEAM-1)

Development of an Autonomous, Explainable, Robust Robotic System for Electric Vehicle Battery Disassembly.

Immersive Assistance System for Intuitive Robot Programming using Mixed-Reality and Digital Twin

AR-based interaction for safe human-robot collaborative manufacturing

An Augmented Reality Interface for Teleoperating Robot Manipulators: Reducing Demonstrator Task Load through Digital Twin Control

Using Augmented Reality to Assess and Modify Mobile Manipulator Surface Repair Plans

Design of a Hyper-Redundant Robot and Teleoperation Using Mixed Reality for Inspection Tasks

A novel assembly process guidance using augmented reality for a standalone hybrid energy system