Abstract:This work introduces a perspective-corrected video see-through mixed-reality head-mounted display with edge-preserving occlusion and low-latency capabilities. To realize the consistent spatial and temporal composition of a captured real world containing virtual objects, we perform three essential tasks: 1) to reconstruct captured images so as to match the user's view; 2) to occlude virtual objects with nearer real objects, to provide users with correct depth cues; and 3) to reproject the virtual and captured scenes to be matched and to keep up with users' head motions. Captured image reconstruction and occlusion-mask generation require dense and accurate depth maps. However, estimating these maps is computationally difficult, which results in longer latencies. To obtain an acceptable balance between spatial consistency and low latency, we rapidly generated depth maps by focusing on edge smoothness and disocclusion (instead of fully accurate maps), to shorten the processing time. Our algorithm refines edges via a hybrid method involving infrared masks and color-guided filters, and it fills disocclusions using temporally cached depth maps. Our system combines these algorithms in a two-phase temporal warping architecture based upon synchronized camera pairs and displays. The first phase of warping is to reduce registration errors between the virtual and captured scenes. The second is to present virtual and captured scenes that correspond with the user's head motion. We implemented these methods on our wearable prototype and performed end-to-end measurements of its accuracy and latency. We achieved an acceptable latency due to head motion (less than 4 ms) and spatial accuracy (less than 0.1° in size and less than 0.3° in position) in our test environment. We anticipate that this work will help improve the realism of mixed reality systems.

Head-worn mixed reality projection display application

Head-mounted mixed reality projection display for games production and entertainment

Development of a Visual Reality Headset and Its Applications in 3D Interactive Bike Race Games

Augment Yourself: Mixed Reality Self-Augmentation Using Optical See-through Head-mounted Displays and Physical Mirrors

Development of Head-Mounted Projection Displays for Distributed, Collaborative, Augmented Reality Applications

A Real-Time Interactive System of Surface Reconstruction and Dynamic Projection Mapping with RGB-depth Sensor and Projector.

Mobile3DRecon: Real-time Monocular 3D Reconstruction on a Mobile Phone

Building Virtual Entertainment Environment with Tiled Display Wall and Motion Tracking.

Facial performance sensing head-mounted display

Integrating Both Parallax and Latency Compensation into Video See-through Head-mounted Display

RePro3D: full-parallax 3D display with haptic feedback using retro-reflective projection technology

6.1: Autostereoscopic Display with a Multidirectional Backlight System and an Eye Tracker

Augmented reality using personal projection and retroreflection

Beaming Displays

ViewR: Architectural-Scale Multi-User Mixed Reality With Mobile Head-Mounted Displays

Inside-out Infrared Marker Tracking via Head Mounted Displays for Smart Robot Programming

A 3D Mixed Reality Interface for Human-Robot Teaming

Retinal 3D

Lightweight Spatial-Multiplexed Dual Focal-Plane Head-Mounted Display Using Two Freeform Prisms

35.1: A novel in‐situ interactive 3D floating autostereoscopic display system with aerial imaging plate

Application of Physical Interactive Mixed Reality System Based on MLAT in the Field of Stage Performance