Abstract:<p>Robotic teleoperation, i.e., manipulating remote robotic systems at a distance, has gained its popularity in various industrial applications, including construction operations. The key to a successful teleoperation robot system is the delicate design of the human-robot interface that helps strengthen the human operator's situational awareness. Traditional human-robot interface for robotic teleoperation is usually based on imagery data (e.g., video streaming), causing the limited field of view (FOV) and increased cognitive burden for processing additional spatial information. As a result, 3D scene reconstruction methods based on point cloud models captured by scanning technologies (e.g., depth camera and LiDAR) have been explored to provide immersive and intuitive feedback to the human operator. Despite the added benefits of applying reconstructed 3D scenes in telerobotic systems, challenges still present. Most 3D reconstruction methods utilize raw point cloud data due to the difficulty of real-time model rendering. The significant size of point cloud data makes the processing and transfer between robots and human operators difficult and slow. In addition, most reconstructed point cloud models do not contain physical properties such as weight and colliders. A more enriched control mechanism based on physics engine simulations is impossible. This paper presents an intelligent robot teleoperation interface that collects, processes, transfers, and reconstructs the immersive scene model of the workspace in Virtual Reality (VR) and enables intuitive robot controls accordingly. The proposed system, Telerobotic Operation based on Auto-reconstructed Remote Scene (TOARS), utilizes a deep learning algorithm to automatically detect objects in the captured scene, along with their physical properties, based on the point cloud data. The processed information is then transferred to the game engine where rendered virtual objects replace the original point cloud models in the VR environment. TOARS is expected to significantly improve the efficiency of 3D scene reconstruction and situational awareness of human operators in robotic teleoperation.</p>

Depth Camera Based Remote Three-Dimensional Reconstruction Using Incremental Point Cloud Compression.

3-D Map Reconstruction in Cross-country Environment

Visual Odometry Based 3D-Reconstruction

Outdoor Scene 3D Reconstruction from Multiple Point Cloud

Remote Dynamic Three-Dimensional Scene Reconstruction

Online Global Non-rigid Registration for 3D Object Reconstruction Using Consumer-level Depth Cameras

Multi-View Depth Map Sampling for 3D Reconstruction of Natural Scene

3D Reconstruction Approach for Outdoor Scene Based on Multiple Point Cloud Fusion

Robust 3D Reconstruction with an RGB-D Camera

Real-time 3D Scene Reconstruction with Dynamically Moving Object Using a Single Depth Camera

Outdoor 3D Environment Reconstruction based on Multi-sensor Fusion for Remote Control

3D Reconstruction of Indoor Scenes Via Image Registration

3d Reconstruction Of Dynamic Scenes With Multiple Handheld Cameras

Virtual Reality Aided High-Quality 3D Reconstruction by Remote Drones

Indoor 3D Scene Reconstruction for Mobile Robots Using Microsoft Kinect Sensor

A Highly Robust Automatic 3D Reconstruction System Based on Integrated Optimization by Point Line Features

Intuitive robot teleoperation for civil engineering operations with virtual reality and deep learning scene reconstruction

Preproessing and Modeling for Visual-Based 3D Indoor Scene Reconstruction

3D Reconstruction System for Collaborative Scanning Based on Multiple RGB-D Cameras.

3D environmental mapping of mobile robot using a low-cost depth camera

Real-time High-accuracy Three-Dimensional Reconstruction with Consumer RGB-D Cameras