Mitigating Motion Blur in Neural Radiance Fields with Events and Frames

Marco Cannici,Davide Scaramuzza

2024-06-04

Abstract:Neural Radiance Fields (NeRFs) have shown great potential in novel view synthesis. However, they struggle to render sharp images when the data used for training is affected by motion blur. On the other hand, event cameras excel in dynamic scenes as they measure brightness changes with microsecond resolution and are thus only marginally affected by blur. Recent methods attempt to enhance NeRF reconstructions under camera motion by fusing frames and events. However, they face challenges in recovering accurate color content or constrain the NeRF to a set of predefined camera poses, harming reconstruction quality in challenging conditions. This paper proposes a novel formulation addressing these issues by leveraging both model- and learning-based modules. We explicitly model the blur formation process, exploiting the event double integral as an additional model-based prior. Additionally, we model the event-pixel response using an end-to-end learnable response function, allowing our method to adapt to non-idealities in the real event-camera sensor. We show, on synthetic and real data, that the proposed approach outperforms existing deblur NeRFs that use only frames as well as those that combine frames and events by +6.13dB and +2.48dB, respectively.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The paper aims to address the issue of Neural Radiance Fields (NeRF) being unable to generate clear images when the training data is affected by motion blur. Specifically, when using images with motion blur for training, existing NeRF methods struggle to reconstruct clear radiance fields, thereby limiting their application in real-world scenarios.

To solve this problem, the authors propose a new method called Ev-DeblurNeRF, which combines data from event cameras and traditional frame data to improve reconstruction quality. Event cameras can capture brightness changes with microsecond-level time resolution and are almost unaffected by motion blur. By integrating event data and frame data, Ev-DeblurNeRF can more accurately recover sharp radiance fields and outperforms existing deblurring NeRF methods on both synthetic and real data.

The specific contributions include:
1. **New deblurring NeRF method**: A new method combining model priors and learning modules is proposed to recover clear NeRF in the presence of motion blur.
2. **Performance improvement**: Experiments show that this method improves by 6.13dB over methods using only frame data and by 2.48dB over methods using both frame and event data.
3. **New datasets**: Two new datasets are introduced, one being a simulated dataset and the other a real-world dataset collected using a Color DAVIS event camera.

With these improvements, Ev-DeblurNeRF can recover textures and fine-grained details in fast-moving scenes, achieving higher reconstruction quality and better performance.