Abstract:An accurate understanding of omnidirectional environment lighting is crucial for high-quality virtual object rendering in mobile augmented reality (AR). In particular, to support reflective rendering, existing methods have leveraged deep learning models to estimate or have used physical light probes to capture physical lighting, typically represented in the form of an environment map. However, these methods often fail to provide visually coherent details or require additional setups. For example, the commercial framework ARKit uses a convolutional neural network that can generate realistic environment maps; however the corresponding reflective rendering might not match the physical environments. In this work, we present the design and implementation of a lighting reconstruction framework called LitAR that enables realistic and visually-coherent rendering. LitAR addresses several challenges of supporting lighting information for mobile AR. First, to address the spatial variance problem, LitAR uses two-field lighting reconstruction to divide the lighting reconstruction task into the spatial variance-aware near-field reconstruction and the directional-aware far-field reconstruction. The corresponding environment map allows reflective rendering with correct color tones. Second, LitAR uses two noise-tolerant data capturing policies to ensure data quality, namely guided bootstrapped movement and motion-based automatic capturing. Third, to handle the mismatch between the mobile computation capability and the high computation requirement of lighting reconstruction, LitAR employs two novel real-time environment map rendering techniques called multi-resolution projection and anchor extrapolation. These two techniques effectively remove the need of time-consuming mesh reconstruction while maintaining visual quality.

Affine-Representation-Based Calibration-Free Augmented Reality Using Image-Based Rendering

A Framework for Lighting Detection and Scene Rendering in AR Environment

Hybrid Meta-Optics Enabled Compact Augmented Reality Display with Computational Image Reinforcement

Real-Time Lighting Estimation for Augmented Reality Via Differentiable Screen-Space Rendering

High-Quality Real-Time Augmented Reality with Automatic Environment Map

LitAR: Visually Coherent Lighting for Mobile Augmented Reality

Multi-Camera Lighting Estimation for Photorealistic Front-Facing Mobile Augmented Reality

A Lighting Consistency Technique for Outdoor Augmented Reality Systems Based on Multi-Source Geo-Information.

Light Re-Rendering Technology Based on Image Brightness Ratio

A New Approach to Outdoor Illumination Estimation Based on Statistical Analysis for Augmented Reality

Camera Calibration In Augmented Reality

A touchable virtual screen interaction system with handheld Kinect camera

Computer Graphics in China: A practical approach for real-time illumination estimation of outdoor videos

Illumination Method in Augmented Reality Based on Dynamic Light Map

Scene-Planning System Based on Augmented Reality Technique

A Mobile Augmented Reality System for Illumination Consistency with Fisheye Camera Based on Client-Server Model

Recovering Pose and Occlusion Consistencies in Augmented Reality Systems Using Affine Properties

Affine Reprojection Based Registration Method for Augmented Reality

Raycast Calibration for Augmented Reality HMDs with Off-Axis Reflective Combiners

IllumiNet: Transferring Illumination from Planar Surfaces to Virtual Objects in Augmented Reality