Towards Physically-Based Sky-Modeling

Ian J. Maquignaz

2024-12-16

Abstract:Accurate environment maps are a key component in rendering photorealistic outdoor scenes with coherent illumination. They enable captivating visual arts, immersive virtual reality and a wide range of engineering and scientific applications. Recent works have extended sky-models to be more comprehensive and inclusive of cloud formations but existing approaches fall short in faithfully recreating key-characteristics in physically captured HDRI. As we demonstrate, environment maps produced by sky-models do not relight scenes with the same tones, shadows, and illumination coherence as physically captured HDR imagery. Though the visual quality of DNN-generated LDR and HDR imagery has greatly progressed in recent years, we demonstrate this progress to be tangential to sky-modelling. Due to the Extended Dynamic Range (EDR) of 14EV required for outdoor environment maps inclusive of the sun, sky-modelling extends beyond the conventional paradigm of High Dynamic Range Imagery (HDRI). In this work, we propose an all-weather sky-model, learning weathered-skies directly from physically captured HDR imagery. Per user-controlled positioning of the sun and cloud formations, our model (AllSky) allows for emulation of physically captured environment maps with improved retention of the Extended Dynamic Range (EDR) of the sky.

Computer Vision and Pattern Recognition,Image and Video Processing

What problem does this paper attempt to address?

The key problem that this paper attempts to solve is how to generate physically - based environment maps that match real - world lighting, especially how to preserve the Extended Dynamic Range (EDR) in these maps. Specifically, the paper focuses on the following aspects:

1. **Limitations of existing models**: Existing sky models are insufficient in reproducing the key characteristics of physically - captured high - dynamic - range images (HDR imagery). For example, they are unable to re - illuminate the scene with the same hue, shadow, and lighting consistency.

2. **Challenges of dynamic range**: Outdoor environment maps need to cover an Extended Dynamic Range of 14EV (Exposure Value) including the sun. However, traditional High - Dynamic - Range Imaging (HDRI) techniques have difficulty meeting this requirement.

3. **Lighting quality**: Although deep neural network (DNN) - generated low - dynamic - range (LDR) and high - dynamic - range (HDR) images have made progress in visual quality, these advances are not directly relevant to sky modeling. Environment maps generated by DNNs are still inferior to physically - captured HDR images in lighting effects.

To solve these problems, the paper proposes a new All - Sky model, which directly learns variable sky states from physically - captured HDR images. By allowing users to control the sun position and cloud morphology, AllSky can more realistically simulate physically - captured environment maps and better preserve the Extended Dynamic Range (EDR).

### Specific contributions

- **Improved dynamic - range preservation**: By introducing intensity - aware and category - aware Selective and Cascade losses, as well as the ANN LDR2HDR enhancer, the paper shows that the generated environment maps have a significant improvement in dynamic range and lighting effects.

- **User - friendly input mode**: Allows users to control the sun position and cloud morphology, thereby achieving precise control of the lighting effects of outdoor scenes.

- **Hybrid μ - lawLog2 tone mapper**: Proposes a hybrid μ - lawLog2 tone mapper for better preservation of texture - rich cloud and sky - dome components while compressing high - luminance areas.

Through these improvements, the paper aims to improve the lighting quality and visual realism of generated environment maps, especially when dealing with the Extended Dynamic Range.