Abstract:This work introduces a perspective-corrected video see-through mixed-reality head-mounted display with edge-preserving occlusion and low-latency capabilities. To realize the consistent spatial and temporal composition of a captured real world containing virtual objects, we perform three essential tasks: 1) to reconstruct captured images so as to match the user's view; 2) to occlude virtual objects with nearer real objects, to provide users with correct depth cues; and 3) to reproject the virtual and captured scenes to be matched and to keep up with users' head motions. Captured image reconstruction and occlusion-mask generation require dense and accurate depth maps. However, estimating these maps is computationally difficult, which results in longer latencies. To obtain an acceptable balance between spatial consistency and low latency, we rapidly generated depth maps by focusing on edge smoothness and disocclusion (instead of fully accurate maps), to shorten the processing time. Our algorithm refines edges via a hybrid method involving infrared masks and color-guided filters, and it fills disocclusions using temporally cached depth maps. Our system combines these algorithms in a two-phase temporal warping architecture based upon synchronized camera pairs and displays. The first phase of warping is to reduce registration errors between the virtual and captured scenes. The second is to present virtual and captured scenes that correspond with the user's head motion. We implemented these methods on our wearable prototype and performed end-to-end measurements of its accuracy and latency. We achieved an acceptable latency due to head motion (less than 4 ms) and spatial accuracy (less than 0.1° in size and less than 0.3° in position) in our test environment. We anticipate that this work will help improve the realism of mixed reality systems.

[Paper] Holographic Video Communication System Realizing Virtual Image Projection and Frontal Image Capture

Holographic near-eye display system based on double-convergence light Gerchberg-Saxton algorithm.

Computer-generated holographic near-eye display system based on LCoS phase only modulator

Real-Time 3D Reconstruction Method for Holographic Telepresence

Development of a Three-Dimensional Multimode Visual Immersive System With Applications in Telepresence

Wide visual field angle holographic display using compact electro-holographic projectors

Slim-panel holographic video display

36‐1: 360‐degree Transparent Light Field Display with Highly‐Directional Holographic Screens for Fully Volumetric 3D Video Experience

Tele-Aloha: A Low-budget and High-authenticity Telepresence System Using Sparse RGB Cameras

[Paper] Volume Holographic Waveguide Using Multiplex Recording for Head-Mounted Display

High-Quality Holographic 3D Display System Based on Virtual Splicing of Spatial Light Modulator

General-purpose Telepresence with Head-Worn Optical See-Through Displays and Projector-Based Lighting

360 Degree Realistic 3D Image Display and Image Processing from Real Objects

Tele-Aloha: A Telepresence System with Low-budget and High-authenticity Using Sparse RGB Cameras

Integrating Both Parallax and Latency Compensation into Video See-through Head-mounted Display

Research on Holographic Communication System Based on 3D Face Reconstruction

Time-multiplexed Neural Holography: A flexible framework for holographic near-eye displays with fast heavily-quantized spatial light modulators

Low-cost, High-Precision Integral 3D Photography and Holographic 3D Display for Real-World Scenes

Compact Lensless Full-Color Holographic Projection System with Digital Phase

Compact Full-Color Holographic 3-D Display Based on Undersampled Computer-Generated Holograms and Oblique Projection Imaging.

48‐4: Invited Paper: High‐resolution Integral Imaging 3D Display System