Abstract:3D Gaussian Splatting has recently emerged as a powerful representation that can synthesize remarkable novel views using consistent multi-view images as input. However, we notice that images captured in dark environments where the scenes are not fully illuminated can exhibit considerable brightness variations and multi-view inconsistency, which poses great challenges to 3D Gaussian Splatting and severely degrades its performance. To tackle this problem, we propose Gaussian-DK. Observing that inconsistencies are mainly caused by camera imaging, we represent a consistent radiance field of the physical world using a set of anisotropic 3D Gaussians, and design a camera response module to compensate for multi-view inconsistencies. We also introduce a step-based gradient scaling strategy to constrain Gaussians near the camera, which turn out to be floaters, from splitting and cloning. Experiments on our proposed benchmark dataset demonstrate that Gaussian-DK produces high-quality renderings without ghosting and floater artifacts and significantly outperforms existing methods. Furthermore, we can also synthesize light-up images by controlling exposure levels that clearly show details in shadow areas.
What problem does this paper attempt to address?
The paper attempts to address the issue of poor view synthesis results caused by inconsistencies in multi-view images taken in low-light environments. Specifically, due to insufficient lighting in low-light environments, there are significant brightness differences and multi-view inconsistencies between photos taken from different angles, which pose a huge challenge to existing 3D Gaussian Splatting technology, severely affecting its performance. The paper proposes a new method—Gaussian-DK, which aims to generate high-quality novel views by addressing these inconsistencies and maintaining high resolution during real-time rendering.
### Main Contributions:
1. **Addressing Multi-View Inconsistencies in Low-Light Environments**: Gaussian-DK can recover consistent scenes from inconsistent images taken in low-light environments.
2. **Synthesizing High Dynamic Range Images**: By controlling exposure levels, Gaussian-DK can also synthesize high dynamic range images, clearly displaying details in shadow areas.
3. **Comprehensive Camera Response Modeling**: A comprehensive camera response modeling pipeline is designed, including exposure level conditions, learnable Gaussian light features, and a step-based gradient scaling strategy to suppress floating objects near the camera.
4. **Proposing a New Benchmark Dataset**: A new benchmark dataset containing 12 real-world scenes is collected to evaluate view synthesis performance in low-light environments.
### Method Overview:
- **3D Gaussian Radiance Field Representation**: A set of 3D Gaussian distributions is used to represent the radiance field of the physical world, with each Gaussian distribution parameterized by mean position, covariance matrix, opacity, and radiance values encoded by spherical harmonics.
- **Camera Response Modeling**: A camera response module is introduced to compensate for inconsistencies in the camera imaging process through exposure level conditions, learnable Gaussian light features, and tone mapping.
- **Floating Object Removal**: A step-based gradient scaling strategy is used to suppress floating objects near the camera, preventing them from splitting or cloning during adaptive density control.
### Experimental Results:
- **Quantitative Comparison**: On the proposed benchmark dataset, Gaussian-DK significantly outperforms other baseline methods in metrics such as Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index (SSIM), and Learned Perceptual Image Patch Similarity (LPIPS).
- **Qualitative Analysis**: Gaussian-DK can generate high-quality novel views, especially excelling in handling complex scenes in low-light environments, effectively reducing artifacts such as floating objects and ghosting.
In summary, Gaussian-DK provides an effective solution for view synthesis in low-light environments, not only improving the quality of synthesized images but also maintaining real-time rendering capabilities.