Deblurring Neural Radiance Fields with Event-driven Bundle Adjustment

Yunshan Qi,Lin Zhu,Yifan Zhao,Nan Bao,Jia Li

DOI: https://doi.org/10.1145/3664647.3680569

2024-08-02

Abstract:Neural Radiance Fields (NeRF) achieves impressive 3D representation learning and novel view synthesis results with high-quality multi-view images as input. However, motion blur in images often occurs in low-light and high-speed motion scenes, which significantly degrades the reconstruction quality of NeRF. Previous deblurring NeRF methods struggle to estimate pose and lighting changes during the exposure time, making them unable to accurately model the motion blur. The bio-inspired event camera measuring intensity changes with high temporal resolution makes up this information deficiency. In this paper, we propose Event-driven Bundle Adjustment for Deblurring Neural Radiance Fields (EBAD-NeRF) to jointly optimize the learnable poses and NeRF parameters by leveraging the hybrid event-RGB data. An intensity-change-metric event loss and a photo-metric blur loss are introduced to strengthen the explicit modeling of camera motion blur. Experiments on both synthetic and real-captured data demonstrate that EBAD-NeRF can obtain accurate camera trajectory during the exposure time and learn a sharper 3D representations compared to prior works.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The paper attempts to address the issue of degraded Neural Radiance Field (NeRF) reconstruction quality caused by camera motion blur in low-light or high-speed motion scenarios. Specifically, existing deblurring NeRF methods struggle to accurately estimate camera poses and lighting changes during exposure time, making it difficult to effectively model motion blur. To solve this problem, the authors propose an Event-Based Bundle Adjustment Deblurring Neural Radiance Field (EBAD-NeRF) framework, which combines event camera data and RGB image data to jointly optimize learnable poses and NeRF parameters for more accurate motion trajectory estimation and clearer 3D representation.

### Main Contributions:
1. **Proposed a novel Event-Based Bundle Adjustment Deblurring Neural Radiance Field (EBAD-NeRF) framework**, which explicitly models image blur and jointly optimizes the estimated camera motion trajectories and NeRF parameters.
2. **Introduced intensity change event loss and photometric blur loss**, supervising the modeling of camera pose changes and motion blur during the image formation process.
3. **Experimental validation on extended synthetic and real datasets** demonstrates that the method can achieve accurate motion trajectory estimation and high-quality 3D implicit scene reconstruction, especially in cases of severely blurred images and corresponding event data.

### Method Overview:
- **Motion blur formation in static 3D scenes**: By modeling the camera pose changes during exposure time as a function of time, the formation process of blurred images is derived.
- **Event-based bundle adjustment**: Utilizes event camera data as a bridge to connect camera pose changes and neural radiance fields, optimizing camera poses by minimizing the difference between predicted and actual events.
- **Final loss function**: Combines photometric blur loss and event loss to supervise the network in learning accurate camera motion trajectories and clear 3D implicit representations.

### Experimental Results:
- **Quantitative analysis**: Experimental results on the Blender synthetic dataset and real datasets show that EBAD-NeRF outperforms existing deblurring NeRF methods in terms of motion trajectory estimation and 3D reconstruction quality.
- **Qualitative analysis**: The generated rendering results are clearer, and the motion trajectory estimation is closer to the ground truth.

In summary, the proposed method effectively addresses the NeRF reconstruction problem in low-light and high-speed motion scenarios by combining event camera data and RGB image data, improving the quality and clarity of 3D scene reconstruction.