Abstract:This work introduces a perspective-corrected video see-through mixed-reality head-mounted display with edge-preserving occlusion and low-latency capabilities. To realize the consistent spatial and temporal composition of a captured real world containing virtual objects, we perform three essential tasks: 1) to reconstruct captured images so as to match the user's view; 2) to occlude virtual objects with nearer real objects, to provide users with correct depth cues; and 3) to reproject the virtual and captured scenes to be matched and to keep up with users' head motions. Captured image reconstruction and occlusion-mask generation require dense and accurate depth maps. However, estimating these maps is computationally difficult, which results in longer latencies. To obtain an acceptable balance between spatial consistency and low latency, we rapidly generated depth maps by focusing on edge smoothness and disocclusion (instead of fully accurate maps), to shorten the processing time. Our algorithm refines edges via a hybrid method involving infrared masks and color-guided filters, and it fills disocclusions using temporally cached depth maps. Our system combines these algorithms in a two-phase temporal warping architecture based upon synchronized camera pairs and displays. The first phase of warping is to reduce registration errors between the virtual and captured scenes. The second is to present virtual and captured scenes that correspond with the user's head motion. We implemented these methods on our wearable prototype and performed end-to-end measurements of its accuracy and latency. We achieved an acceptable latency due to head motion (less than 4 ms) and spatial accuracy (less than 0.1° in size and less than 0.3° in position) in our test environment. We anticipate that this work will help improve the realism of mixed reality systems.

General-purpose Telepresence with Head-Worn Optical See-Through Displays and Projector-Based Lighting

Development of a Three-Dimensional Multimode Visual Immersive System With Applications in Telepresence

Tele-Aloha: A Low-budget and High-authenticity Telepresence System Using Sparse RGB Cameras

Development of a Visual Reality Headset and Its Applications in 3D Interactive Bike Race Games

Tele-Aloha: A Telepresence System with Low-budget and High-authenticity Using Sparse RGB Cameras

TeleHuman2

Bringing Telepresence to Every Desk

Real-time stereoscopic rendering of realistic avatar for interactive 3D telepresence system

Viewpoint-Controllable Telepresence: A Robotic-Arm-Based Mixed-Reality Telecollaboration System

Foveated Light-Field Display and Real-Time Rendering for Virtual Reality

Low-cost telepresence for collaborative virtual environments

33‐1: Invited Paper: Tabletop True 3D Display Systems Based on Projection Light Field and Integral Imaging

Tele-AR System Based on Real-time Camera Tracking

Wearable Optical See-Through Head-Mounted Display Capable of Adjusting Virtual Image Depth

A Multi-Plane Optical See-Through Head Mounted Display Design for Augmented Reality Applications

Virtual Reality Based Immersive Telepresence System for Remote Conversation and Collaboration.

Integrating Both Parallax and Latency Compensation into Video See-through Head-mounted Display

RemoteTouch: Enhancing Immersive 3D Video Communication with Hand Touch

An Optical See-Through Head Mounted Display with Addressable Focal Planes

A Robotic Arm-based Telepresence for Mixed-Reality Telecollaboration System.

Low-Complexity 3D-Vision Conferencing System Based on Accelerated RIFE Model