Abstract:This work introduces a perspective-corrected video see-through mixed-reality head-mounted display with edge-preserving occlusion and low-latency capabilities. To realize the consistent spatial and temporal composition of a captured real world containing virtual objects, we perform three essential tasks: 1) to reconstruct captured images so as to match the user's view; 2) to occlude virtual objects with nearer real objects, to provide users with correct depth cues; and 3) to reproject the virtual and captured scenes to be matched and to keep up with users' head motions. Captured image reconstruction and occlusion-mask generation require dense and accurate depth maps. However, estimating these maps is computationally difficult, which results in longer latencies. To obtain an acceptable balance between spatial consistency and low latency, we rapidly generated depth maps by focusing on edge smoothness and disocclusion (instead of fully accurate maps), to shorten the processing time. Our algorithm refines edges via a hybrid method involving infrared masks and color-guided filters, and it fills disocclusions using temporally cached depth maps. Our system combines these algorithms in a two-phase temporal warping architecture based upon synchronized camera pairs and displays. The first phase of warping is to reduce registration errors between the virtual and captured scenes. The second is to present virtual and captured scenes that correspond with the user's head motion. We implemented these methods on our wearable prototype and performed end-to-end measurements of its accuracy and latency. We achieved an acceptable latency due to head motion (less than 4 ms) and spatial accuracy (less than 0.1° in size and less than 0.3° in position) in our test environment. We anticipate that this work will help improve the realism of mixed reality systems.

See-Through Face Display: Enabling Gaze Communication for Any Face$\unicode{x2013}$Human or AI

Social Glasses: Simulating Interactive Gaze for Visually Impaired People in Face-to-Face Communication

Air Mounted Eyepiece: Design Methods for Aerial Optical Functions of Near-Eye and See-Through Display using Transmissive Mirror Device

See-Through Captions: Real-Time Captioning on Transparent Display for Deaf and Hard-of-Hearing People

Facial performance sensing head-mounted display

Real-Time 3D Face Reconstruction and Gaze Tracking for Virtual Reality

Instant interaction driven adaptive gaze control interface

Integrating Both Parallax and Latency Compensation into Video See-through Head-mounted Display

FaceLooks: A Smart Headband for Signaling Face-to-Face Behavior

[Paper] Holographic Video Communication System Realizing Virtual Image Projection and Frontal Image Capture

Eye gaze correction with stereovision for video-teleconferencing

3D Face Reconstruction and Gaze Tracking in the HMD for Virtual Interaction `

Wearable Optical See-Through Head-Mounted Display Capable of Adjusting Virtual Image Depth

The Effect of Eye Contact in Multi-Party Conversations with Virtual Humans and Mitigating the Mona Lisa Effect

Real-time Face View Correction for Front-Facing Cameras

Design and implementation of an optical see-through near-eye display combining Maxwellian-view and light-field methods

Realtime Dynamic Gaze Target Tracking and Depth-Level Estimation

eyemR-Vis: A Mixed Reality System to Visualise Bi-Directional Gaze Behavioural Cues Between Remote Collaborators

An Optical See-Through Augmented Reality System with Gesture-Based Interaction

General-purpose Telepresence with Head-Worn Optical See-Through Displays and Projector-Based Lighting

Virtual Reflections on a Dynamic 2D Eye Model Improve Spatial Reference Identification