Abstract:An accurate understanding of omnidirectional environment lighting is crucial for high-quality virtual object rendering in mobile augmented reality (AR). In particular, to support reflective rendering, existing methods have leveraged deep learning models to estimate or have used physical light probes to capture physical lighting, typically represented in the form of an environment map. However, these methods often fail to provide visually coherent details or require additional setups. For example, the commercial framework ARKit uses a convolutional neural network that can generate realistic environment maps; however the corresponding reflective rendering might not match the physical environments. In this work, we present the design and implementation of a lighting reconstruction framework called LitAR that enables realistic and visually-coherent rendering. LitAR addresses several challenges of supporting lighting information for mobile AR. First, to address the spatial variance problem, LitAR uses two-field lighting reconstruction to divide the lighting reconstruction task into the spatial variance-aware near-field reconstruction and the directional-aware far-field reconstruction. The corresponding environment map allows reflective rendering with correct color tones. Second, LitAR uses two noise-tolerant data capturing policies to ensure data quality, namely guided bootstrapped movement and motion-based automatic capturing. Third, to handle the mismatch between the mobile computation capability and the high computation requirement of lighting reconstruction, LitAR employs two novel real-time environment map rendering techniques called multi-resolution projection and anchor extrapolation. These two techniques effectively remove the need of time-consuming mesh reconstruction while maintaining visual quality.

A Mobile Augmented Reality System for Illumination Consistency with Fisheye Camera Based on Client-Server Model

An Off-Screen Object Visualization Approach for Mobile Augment Reality

Cloud-based Fusion Rendering for AR Applications

A Framework for Lighting Detection and Scene Rendering in AR Environment

A Lighting Consistency Technique for Outdoor Augmented Reality Systems Based on Multi-Source Geo-Information.

A Visual-Inertial Fusion Based Tracking System For Mobile Augmented Reality

Affine-Representation-Based Calibration-Free Augmented Reality Using Image-Based Rendering

Image-Based Walkthrough over Internet on Mobile Devices.

Lighting Simulation of Augmented Outdoor Scene Based on a Legacy Photograph.

Multi-Camera Lighting Estimation for Photorealistic Front-Facing Mobile Augmented Reality

Scene-Planning System Based on Augmented Reality Technique

Mobile3DRecon: Real-time Monocular 3D Reconstruction on a Mobile Phone

Real-Time Lighting Estimation for Augmented Reality Via Differentiable Screen-Space Rendering

High-Quality Real-Time Augmented Reality with Automatic Environment Map

Efficient mobile AR technology using scalable recognition and tracking based on server-client model

A New Approach to Outdoor Illumination Estimation Based on Statistical Analysis for Augmented Reality

PanoWalk: a remote image-based rendering system for mobile devices

LitAR: Visually Coherent Lighting for Mobile Augmented Reality

A touchable virtual screen interaction system with handheld Kinect camera

On-line Illumination Estimation of Outdoor Scenes Based on Area Selection for Augmented Reality

Xihe: A 3D Vision-based Lighting Estimation Framework for Mobile Augmented Reality