Abstract:This work introduces a perspective-corrected video see-through mixed-reality head-mounted display with edge-preserving occlusion and low-latency capabilities. To realize the consistent spatial and temporal composition of a captured real world containing virtual objects, we perform three essential tasks: 1) to reconstruct captured images so as to match the user's view; 2) to occlude virtual objects with nearer real objects, to provide users with correct depth cues; and 3) to reproject the virtual and captured scenes to be matched and to keep up with users' head motions. Captured image reconstruction and occlusion-mask generation require dense and accurate depth maps. However, estimating these maps is computationally difficult, which results in longer latencies. To obtain an acceptable balance between spatial consistency and low latency, we rapidly generated depth maps by focusing on edge smoothness and disocclusion (instead of fully accurate maps), to shorten the processing time. Our algorithm refines edges via a hybrid method involving infrared masks and color-guided filters, and it fills disocclusions using temporally cached depth maps. Our system combines these algorithms in a two-phase temporal warping architecture based upon synchronized camera pairs and displays. The first phase of warping is to reduce registration errors between the virtual and captured scenes. The second is to present virtual and captured scenes that correspond with the user's head motion. We implemented these methods on our wearable prototype and performed end-to-end measurements of its accuracy and latency. We achieved an acceptable latency due to head motion (less than 4 ms) and spatial accuracy (less than 0.1° in size and less than 0.3° in position) in our test environment. We anticipate that this work will help improve the realism of mixed reality systems.

Add-on Occlusion: Building an Occlusion-capable Optical See-through Head-mounted Display with HoloLens 1.

Add-on Occlusion: Turning Off-the-Shelf Optical See-through Head-mounted Displays Occlusion-capable

Add-on Occlusion: an External Module for Optical See-through Augmented Reality Displays to Support Mutual Occlusion

Factored Occlusion: Single Spatial Light Modulator Occlusion-capable Optical See-through Augmented Reality Display

Computer-generated holographic near-eye display system based on LCoS phase only modulator

Color Transparent Screen Using Planar Glass Combined with Lens Array Holographic Optical Elements

6‐1: Multi‐dimensional Convertible Optical See‐through Head‐mounted Display Using Multiplexed Holographic Optical Element

A Multi-Plane Optical See-Through Head Mounted Display Design for Augmented Reality Applications

17‐1: A Flat‐panel Holographic‐optical‐element System for Holographic Augmented Reality Display with a Beam Expander

Compact Design for Optical-See-through Holographic Displays Employing Holographic Optical Elements

3-1: A Multi-plane Volumetric Optical See-through Head Mounted 3D Display

41.3: A Flat‐panel Holographic‐optical‐element System for Holographic Augmented Reality Display with a Beam Expander

A Multiplane Volumetric Optical See-through Head Mounted 3 D Display †

Integrating Both Parallax and Latency Compensation into Video See-through Head-mounted Display

P‐51: Student Poster: Color Contrast Enhanced Rendering for Optical See‐Through Head‐Mounted Displays with Optimized Display Power

Foveated Light-Field Display and Real-Time Rendering for Virtual Reality

Varifocal occlusion in optical see-through near-eye display with single phase-only liquid crystal on silicon

Development of a Visual Reality Headset and Its Applications in 3D Interactive Bike Race Games

Design and fabrication of an off-axis see-through head-mounted display with an x-y polynomial surface

Holographic pancake optics for thin and lightweight optical see-through augmented reality.

HoloDevice: Holographic Cross-Device Interactions for Remote Collaboration