Abstract:An accurate understanding of omnidirectional environment lighting is crucial for high-quality virtual object rendering in mobile augmented reality (AR). In particular, to support reflective rendering, existing methods have leveraged deep learning models to estimate or have used physical light probes to capture physical lighting, typically represented in the form of an environment map. However, these methods often fail to provide visually coherent details or require additional setups. For example, the commercial framework ARKit uses a convolutional neural network that can generate realistic environment maps; however the corresponding reflective rendering might not match the physical environments. In this work, we present the design and implementation of a lighting reconstruction framework called LitAR that enables realistic and visually-coherent rendering. LitAR addresses several challenges of supporting lighting information for mobile AR. First, to address the spatial variance problem, LitAR uses two-field lighting reconstruction to divide the lighting reconstruction task into the spatial variance-aware near-field reconstruction and the directional-aware far-field reconstruction. The corresponding environment map allows reflective rendering with correct color tones. Second, LitAR uses two noise-tolerant data capturing policies to ensure data quality, namely guided bootstrapped movement and motion-based automatic capturing. Third, to handle the mismatch between the mobile computation capability and the high computation requirement of lighting reconstruction, LitAR employs two novel real-time environment map rendering techniques called multi-resolution projection and anchor extrapolation. These two techniques effectively remove the need of time-consuming mesh reconstruction while maintaining visual quality.

Review on Illumination Estimation in Augmented Reality

A Framework for Lighting Detection and Scene Rendering in AR Environment

A New Approach to Outdoor Illumination Estimation Based on Statistical Analysis for Augmented Reality

On-line Illumination Estimation of Outdoor Scenes Based on Area Selection for Augmented Reality

Light direction estimation and hand touchable interaction for augmented reality

Scene-Planning System Based on Augmented Reality Technique

Computer Graphics in China: A practical approach for real-time illumination estimation of outdoor videos

Lighting Simulation of Augmented Outdoor Scene Based on a Legacy Photograph.

Online Illumination Estimation of Outdoor Scenes Based on Videos Containing No Shadow Area.

Robust and Real-Time Outdoor Illumination Estimation

53.3: Invited Paper: an Overview of International Standardization for Image Quality of AR Displays

Xihe: A 3D Vision-based Lighting Estimation Framework for Mobile Augmented Reality

LitAR: Visually Coherent Lighting for Mobile Augmented Reality

Real-time Outdoor Illumination Estimation Based on Energy Optimization

Handling Occlusions in Video-Based Augmented Reality Using Depth Information

Survey and Evaluation of Monocular Visual-Inertial SLAM Algorithms for Augmented Reality

A Survey of Monocular Simultaneous Localization and Mapping

Advanced Visual SLAM and Image Segmentation Techniques for Augmented Reality

An Improved Augmented-Reality Framework for Differential Rendering Beyond the Lambertian-World Assumption

Multi-Camera Lighting Estimation for Photorealistic Front-Facing Mobile Augmented Reality

Depth Perception in Optical See-Through Augmented Reality: Investigating the Impact of Texture Density, Luminance Contrast, and Color Contrast