Abstract:Neural Radiance Fields (NeRFs) have shown remarkable performances in producing novel-view images from high-quality scene images. However, hand-held low-light photography challenges NeRFs as the captured images may simultaneously suffer from low visibility, noise, and camera shakes. While existing NeRF methods may handle either low light or motion, directly combining them or incorporating additional image-based enhancement methods does not work as these degradation factors are highly coupled. We observe that noise in low-light images is always sharp regardless of camera shakes, which implies an implicit order of these degradation factors within the image formation process. To this end, we propose in this paper a novel model, named LuSh-NeRF, which can reconstruct a clean and sharp NeRF from a group of hand-held low-light images. The key idea of LuSh-NeRF is to sequentially model noise and blur in the images via multi-view feature consistency and frequency information of NeRF, respectively. Specifically, LuSh-NeRF includes a novel Scene-Noise Decomposition (SND) module for decoupling the noise from the scene representation and a novel Camera Trajectory Prediction (CTP) module for the estimation of camera motions based on low-frequency scene information. To facilitate training and evaluations, we construct a new dataset containing both synthetic and real images. Experiments show that LuSh-NeRF outperforms existing approaches. Our code and dataset can be found here: <a class="link-external link-https" href="https://github.com/quzefan/LuSh-NeRF" rel="external noopener nofollow">this https URL</a>.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is that in photos taken in a hand - held low - light environment, due to the existence of multiple degradation factors such as low visibility, noise, and camera shake, it is difficult for NeRF (Neural Radiance Field) to generate high - quality new - view images. Specifically: 1. **Low Visibility**: Images in low - light scenes usually have low pixel intensities, which makes NeRF training difficult and thus affects the quality of the generated images. 2. **Noise**: Sensor noise inevitably exists during low - light imaging, and this noise is random and unpredictable, which will seriously interfere with the image quality. 3. **Camera Motion Blur**: Long - exposure will inevitably lead to blurring due to camera movement. Existing NeRF methods can either only handle low - light scenes or only handle motion blur, but cannot handle both problems simultaneously. Directly combining existing low - light enhancement or deblurring methods cannot solve the problem well either, because these degradation factors are highly coupled during the image formation process. To solve these problems, the authors proposed the LuSh - NeRF model, which decouples and removes these degradation factors by modeling the implicit order of these degradation factors, thereby achieving the reconstruction of a bright, clear, and sharp NeRF scene from hand - held low - light photos. ### Specific Solutions The core idea of LuSh - NeRF is to model noise and blur in sequence, mainly including two novel modules: 1. **Scene - Noise Decomposition (SND) Module**: - This module separates noise from the scene representation through multi - view feature consistency. - A Noise NeRF architecture is used to decompose the noise in each view, and the multi - view consistency characteristics in the 3D scene are utilized to better separate scene information and noise. 2. **Camera Trajectory Prediction (CTP) Module**: - This module estimates the camera trajectory based on low - frequency scene information to reduce the interference of noise on blur kernel prediction. - The discrete Fourier transform (DFT) is used to obtain the image frequency map, and a low - pass filter is used to filter out high - noise areas, thereby retaining low - frequency information areas for accurate prediction of camera movement. Through the synergy of these two modules, LuSh - NeRF can effectively remove noise and blur during the training process and finally generate high - quality new - view images. ### Experimental Verification To verify the effectiveness of LuSh - NeRF, the authors constructed a new dataset containing synthetic and real - world scenes and conducted a large number of experiments. The experimental results show that LuSh - NeRF can significantly outperform existing methods when processing hand - held low - light photos and generate more natural, clear, and sharp images. ### Summary The main contributions of LuSh - NeRF are: - Proposing the first method that can reconstruct NeRF from hand - held low - light photos by modeling the implicit order of degradation factors to decouple and remove these degradation factors. - Introducing two novel modules: the SND module for noise removal and the CTP module for handling camera movement. - Constructing the first hand - held low - light NeRF dataset for training and evaluation. These innovations make LuSh - NeRF perform well in handling complex low - light scenes and provide a new direction for future research.

LuSh-NeRF: Lighting up and Sharpening NeRFs for Low-light Scenes

Lighting up NeRF via Unsupervised Decomposition and Enhancement

NeRF in the Dark: High Dynamic Range View Synthesis from Noisy Raw Images

Aleth-NeRF: Low-light Condition View Synthesis with Concealing Fields

Ambient-NeRF: light train enhancing neural radiance fields in low-light conditions with ambient-illumination

USB-NeRF: Unrolling Shutter Bundle Adjusted Neural Radiance Fields

RustNeRF: Robust Neural Radiance Field with Low-Quality Images

ZS-SRT: An Efficient Zero-Shot Super-Resolution Training Method for Neural Radiance Fields

BeNeRF: Neural Radiance Fields from a Single Blurry Image and Event Stream

RS-NeRF: Neural Radiance Fields from Rolling Shutter Images

NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections

Aleth-NeRF: Illumination Adaptive NeRF with Concealing Field Assumption

Deblur-NeRF: Neural Radiance Fields from Blurry Images

LSE-NeRF: Learning Sensor Modeling Errors for Deblured Neural Radiance Fields with RGB-Event Stereo

Self-NeRF: A Self-Training Pipeline for Few-Shot Neural Radiance Fields

Clean-NeRF: Reformulating NeRF to account for View-Dependent Observations

Camera Relocalization in Shadow-free Neural Radiance Fields

Sharp-NeRF: Grid-based Fast Deblurring Neural Radiance Fields Using Sharpness Prior

Robust e-NeRF: NeRF from Sparse & Noisy Events under Non-Uniform Motion

SN 2 eRF: A Framework for Neural Radiance Fields given Sparse and Noisy Poses

NeRF On-the-go: Exploiting Uncertainty for Distractor-free NeRFs in the Wild