Abstract:This work introduces a perspective-corrected video see-through mixed-reality head-mounted display with edge-preserving occlusion and low-latency capabilities. To realize the consistent spatial and temporal composition of a captured real world containing virtual objects, we perform three essential tasks: 1) to reconstruct captured images so as to match the user's view; 2) to occlude virtual objects with nearer real objects, to provide users with correct depth cues; and 3) to reproject the virtual and captured scenes to be matched and to keep up with users' head motions. Captured image reconstruction and occlusion-mask generation require dense and accurate depth maps. However, estimating these maps is computationally difficult, which results in longer latencies. To obtain an acceptable balance between spatial consistency and low latency, we rapidly generated depth maps by focusing on edge smoothness and disocclusion (instead of fully accurate maps), to shorten the processing time. Our algorithm refines edges via a hybrid method involving infrared masks and color-guided filters, and it fills disocclusions using temporally cached depth maps. Our system combines these algorithms in a two-phase temporal warping architecture based upon synchronized camera pairs and displays. The first phase of warping is to reduce registration errors between the virtual and captured scenes. The second is to present virtual and captured scenes that correspond with the user's head motion. We implemented these methods on our wearable prototype and performed end-to-end measurements of its accuracy and latency. We achieved an acceptable latency due to head motion (less than 4 ms) and spatial accuracy (less than 0.1° in size and less than 0.3° in position) in our test environment. We anticipate that this work will help improve the realism of mixed reality systems.

Real-time mixed-reality telepresence via 3D reconstruction with HoloLens and commodity depth sensors.

Real-Time 3D Reconstruction Method for Holographic Telepresence

A 3D Mixed Reality Interface for Human-Robot Teaming

A Real-Time Interactive System of Surface Reconstruction and Dynamic Projection Mapping with RGB-depth Sensor and Projector.

Mobile3DRecon: Real-time Monocular 3D Reconstruction on a Mobile Phone

Combining HoloLens with Instant-NeRFs: Advanced Real-Time 3D Mobile Mapping

Monocular Real-Time Human Geometry Reconstruction

General-purpose Telepresence with Head-Worn Optical See-Through Displays and Projector-Based Lighting

Mixed reality human teleoperation with device-agnostic remote ultrasound: Communication and user interaction

Tele-Aloha: A Telepresence System with Low-budget and High-authenticity Using Sparse RGB Cameras

Tele-Aloha: A Low-budget and High-authenticity Telepresence System Using Sparse RGB Cameras

Integrating Both Parallax and Latency Compensation into Video See-through Head-mounted Display

Remote Assistance with Mixed Reality for Procedural Tasks

Mid-Air Haptic Bio-Holograms in Mixed Reality

A DirectX-Based DICOM Viewer for Multi-User Surgical Planning in Augmented Reality

3D Real Human Reconstruction Via Multiple Low-Cost Depth Cameras.

Live Mixed Reality Video Production for Educational Presentation

Real-Time Human Motion Transfer System for Holographic Displays

Real-time cloud-based 3D reconstruction and interaction with a stereo smartphone

An Integrated Platform for Live 3D Human Reconstruction and Motion Capturing

Real-time 3d reconstruction of dynamic scenes with multiple kinect v2 sensors