Lighting Every Darkness with 3DGS: Fast Training and Real-Time Rendering for HDR View Synthesis

Xin Jin,Pengyi Jiao,Zheng-Peng Duan,Xingchao Yang,Chun-Le Guo,Bo Ren,Chongyi Li

2024-06-10

Abstract:Volumetric rendering based methods, like NeRF, excel in HDR view synthesis from RAWimages, especially for nighttime scenes. While, they suffer from long training times and cannot perform real-time rendering due to dense sampling requirements. The advent of 3D Gaussian Splatting (3DGS) enables real-time rendering and faster training. However, implementing RAW image-based view synthesis directly using 3DGS is challenging due to its inherent drawbacks: 1) in nighttime scenes, extremely low SNR leads to poor structure-from-motion (SfM) estimation in distant views; 2) the limited representation capacity of spherical harmonics (SH) function is unsuitable for RAW linear color space; and 3) inaccurate scene structure hampers downstream tasks such as refocusing. To address these issues, we propose LE3D (Lighting Every darkness with 3DGS). Our method proposes Cone Scatter Initialization to enrich the estimation of SfM, and replaces SH with a Color MLP to represent the RAW linear color space. Additionally, we introduce depth distortion and near-far regularizations to improve the accuracy of scene structure for downstream tasks. These designs enable LE3D to perform real-time novel view synthesis, HDR rendering, refocusing, and tone-mapping changes. Compared to previous volumetric rendering based methods, LE3D reduces training time to 1% and improves rendering speed by up to 4,000 times for 2K resolution images in terms of FPS. Code and viewer can be found in <a class="link-external link-https" href="https://github.com/Srameo/LE3D" rel="external noopener nofollow">this https URL</a> .

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The problem that this paper attempts to solve is to quickly train from multi - view noisy RAW images and achieve real - time HDR view synthesis in night - time or high - contrast scenes. Specifically, existing methods such as NeRF, although performing well in HDR view synthesis, have the problems of long training time and inability to render in real - time. Although 3D Gaussian Splatting (3DGS) can achieve fast training and real - time rendering, it faces the following challenges when directly using noisy RAW images for view synthesis:

1. **Low signal - to - noise ratio in night - time scenes**: This leads to inaccurate structure - from - motion (SfM) estimation, especially in long - distance views.
2. **Limited representation ability of spherical harmonics (SH)**: It is not suitable for the RAW linear color space.
3. **Inaccurate scene structure**: This affects the performance of subsequent tasks such as refocusing.

To overcome these challenges, the paper proposes LE3D (Lighting Every darkness with 3DGS), which improves 3DGS through the following methods:

1. **Cone Scatter Initialization**: It enriches the SfM point cloud estimated by COLMAP and improves the accuracy of long - distance views.
2. **Color MLP**: A small MLP is used instead of SH to represent the RAW linear color space, which improves the accuracy and stability of color representation.
3. **Depth distortion and near - far regularization**: By introducing depth distortion regularization \( R_{\text{dist}} \) and near - far regularization \( R_{\text{nf}} \), the accuracy of the scene structure is improved, thereby enhancing the performance of downstream tasks.

These improvements enable LE3D to significantly reduce training time and increase rendering speed while maintaining high - quality view synthesis. Compared with previous volume - rendering - based methods, LE3D reduces training time by 99% and increases rendering speed by 4000 times. In addition, LE3D also supports downstream tasks such as real - time HDR rendering, refocusing, and exposure changes.