Abstract:Omnidirectional lighting provides the foundation for achieving spatially-variant photorealistic 3D rendering, a desirable property for mobile augmented reality applications. However, in practice, estimating omnidirectional lighting can be challenging due to limitations such as partial panoramas of the rendering positions, and the inherent environment lighting and mobile user dynamics. A new opportunity arises recently with the advancements in mobile 3D vision, including built-in high-accuracy depth sensors and deep learning-powered algorithms, which provide the means to better sense and understand the physical surroundings. Centering the key idea of 3D vision, in this work, we design an edge-assisted framework called Xihe to provide mobile AR applications the ability to obtain accurate omnidirectional lighting estimation in real time. Specifically, we develop a novel sampling technique that efficiently compresses the raw point cloud input generated at the mobile device. This technique is derived based on our empirical analysis of a recent 3D indoor dataset and plays a key role in our 3D vision-based lighting estimator pipeline design. To achieve the real-time goal, we develop a tailored GPU pipeline for on-device point cloud processing and use an encoding technique that reduces network transmitted bytes. Finally, we present an adaptive triggering strategy that allows Xihe to skip unnecessary lighting estimations and a practical way to provide temporal coherent rendering integration with the mobile AR ecosystem. We evaluate both the lighting estimation accuracy and time of Xihe using a reference mobile application developed with Xihe's APIs. Our results show that Xihe takes as fast as 20.67ms per lighting estimation and achieves 9.4% better estimation accuracy than a state-of-the-art neural network.

A New Approach to Outdoor Illumination Estimation Based on Statistical Analysis for Augmented Reality

Computer Graphics in China: A practical approach for real-time illumination estimation of outdoor videos

Online Illumination Estimation of Outdoor Scenes Based on Videos Containing No Shadow Area.

On-line Illumination Estimation of Outdoor Scenes Based on Area Selection for Augmented Reality

Robust and Real-Time Outdoor Illumination Estimation

Real-time Outdoor Illumination Estimation Based on Energy Optimization

Lighting Simulation of Augmented Outdoor Scene Based on a Legacy Photograph.

A Framework for Lighting Detection and Scene Rendering in AR Environment

Light source estimation of outdoor scenes for mixed reality

Basis Image Decomposition of Outdoor Time-Lapse Videos.

An Algorithm for Outdoor Illumination Estimation Based on Partial Least Squares Method

Basis Image Decomposition for Outdoor Lighting Estimation

Image-based Relighting from a Sparse Set of Outdoor Images.

Xihe: A 3D Vision-based Lighting Estimation Framework for Mobile Augmented Reality

Illumination Parameter Estimation of Outdoor Scene Using BP Neural Network

High-Quality Real-Time Augmented Reality with Automatic Environment Map

Handling Occlusions in Video-Based Augmented Reality Using Depth Information

Real time stereo rendering for augmented reality on 3DTV system

Multi-Camera Lighting Estimation for Photorealistic Front-Facing Mobile Augmented Reality

A real-time global relighting algorithm

Indoor Lighting Estimation using an Event Camera