Abstract:<p>Robotic teleoperation, i.e., manipulating remote robotic systems at a distance, has gained its popularity in various industrial applications, including construction operations. The key to a successful teleoperation robot system is the delicate design of the human-robot interface that helps strengthen the human operator's situational awareness. Traditional human-robot interface for robotic teleoperation is usually based on imagery data (e.g., video streaming), causing the limited field of view (FOV) and increased cognitive burden for processing additional spatial information. As a result, 3D scene reconstruction methods based on point cloud models captured by scanning technologies (e.g., depth camera and LiDAR) have been explored to provide immersive and intuitive feedback to the human operator. Despite the added benefits of applying reconstructed 3D scenes in telerobotic systems, challenges still present. Most 3D reconstruction methods utilize raw point cloud data due to the difficulty of real-time model rendering. The significant size of point cloud data makes the processing and transfer between robots and human operators difficult and slow. In addition, most reconstructed point cloud models do not contain physical properties such as weight and colliders. A more enriched control mechanism based on physics engine simulations is impossible. This paper presents an intelligent robot teleoperation interface that collects, processes, transfers, and reconstructs the immersive scene model of the workspace in Virtual Reality (VR) and enables intuitive robot controls accordingly. The proposed system, Telerobotic Operation based on Auto-reconstructed Remote Scene (TOARS), utilizes a deep learning algorithm to automatically detect objects in the captured scene, along with their physical properties, based on the point cloud data. The processed information is then transferred to the game engine where rendered virtual objects replace the original point cloud models in the VR environment. TOARS is expected to significantly improve the efficiency of 3D scene reconstruction and situational awareness of human operators in robotic teleoperation.</p>

Research on 3D Remote Virtual Monitoring for Tower Crane

Study on Remote Monitoring System of 3D Scanning and Measuring Machine

CONVR2006 Visualizing Crane Selection and Operation in Virtual Environment

A VR-based the emergency rescue training system of railway accident

Building High Performance DVR Via HLA, Scene Graph and Parallel Rendering.

3D Virtual Studio System Based on Electromechanical Tracking

Design of 3D Scene Visual Communication Modeling Based on Virtual Reality Graphics Rendering Framework

Design of the Virtual Reality-based Remote Equipment Monitoring System

Research on safety monitoring and early warning technology of intelligent auxiliary live working

Vision-Based Automated Recognition and 3D Localization Framework for Tower Cranes Using Far-Field Cameras

Research on the Application of Environmental Art Design Based on the Combination of VR and Panoramic Video Technology

Airport Fire Emergency Rescue Virtual Drill Platform Based on Virtual Reality Technique

A Distributed System Monitoring Tool with Virtual Reality

Simulation Training System for Supervision of Power Work Safety Based on Virtual Reality

Intuitive robot teleoperation for civil engineering operations with virtual reality and deep learning scene reconstruction

Effect of Virtual Annotation on Performance of Construction Equipment Teleoperation under Adverse Visual Conditions

Research on Dynamic Visual Simulation of Hydropower Project Construction Based on Virtual Reality

Research on the Application of Animation Technology in Three Dimensional City Modeling

Research on the Real-Time Mechatronic System Simulation Based on Virtual Reality

VR Panoramic Technology in Urban Rail Transit Vehicle Engineering Simulation System

Real-time 3D unstructured environment reconstruction utilizing VR and Kinect-based immersive teleoperation for agricultural field robots