USB-NeRF: Unrolling Shutter Bundle Adjusted Neural Radiance Fields

Moyang Li,Peng Wang,Lingzhe Zhao,Bangyan Liao,Peidong Liu

2024-02-26

Abstract:Neural Radiance Fields (NeRF) has received much attention recently due to its impressive capability to represent 3D scene and synthesize novel view images. Existing works usually assume that the input images are captured by a global shutter camera. Thus, rolling shutter (RS) images cannot be trivially applied to an off-the-shelf NeRF algorithm for novel view synthesis. Rolling shutter effect would also affect the accuracy of the camera pose estimation (e.g. via COLMAP), which further prevents the success of NeRF algorithm with RS images. In this paper, we propose Unrolling Shutter Bundle Adjusted Neural Radiance Fields (USB-NeRF). USB-NeRF is able to correct rolling shutter distortions and recover accurate camera motion trajectory simultaneously under the framework of NeRF, by modeling the physical image formation process of a RS camera. Experimental results demonstrate that USB-NeRF achieves better performance compared to prior works, in terms of RS effect removal, novel view image synthesis as well as camera motion estimation. Furthermore, our algorithm can also be used to recover high-fidelity high frame-rate global shutter video from a sequence of RS images.

Computer Vision and Pattern Recognition,Artificial Intelligence

What problem does this paper attempt to address?

The paper attempts to address the impact of the Rolling Shutter (RS) effect in Neural Radiance Fields (NeRF). Specifically:

1. **Removing Rolling Shutter Effect**: Existing works usually assume that input images are captured by a global shutter camera, thus rolling shutter images cannot be directly applied to standard NeRF algorithms for novel view synthesis. This paper proposes a new method, USB-NeRF, which can correct rolling shutter distortions within the NeRF framework.

2. **Improving Camera Pose Estimation Accuracy**: The rolling shutter effect affects camera pose estimation (e.g., estimated through COLMAP), which in turn impacts the performance of NeRF algorithms. USB-NeRF can simultaneously recover accurate camera motion trajectories.

3. **Generating High-Quality Global Shutter Videos**: By modeling the actual imaging process of rolling shutter cameras, USB-NeRF can not only remove the rolling shutter effect but also recover high-fidelity global shutter videos from a series of rolling shutter images.

Experimental results show that USB-NeRF outperforms previous works in removing the rolling shutter effect, novel view synthesis, and camera motion estimation, and can generate high-frame-rate global shutter videos. Compared to learning-based methods, USB-NeRF does not require pre-trained datasets, thus offering better generalization performance.