CBARF: Cascaded Bundle-Adjusting Neural Radiance Fields from Imperfect Camera Poses

Hongyu Fu,Xin Yu,Lincheng Li,Li Zhang

2023-10-15

Abstract:Existing volumetric neural rendering techniques, such as Neural Radiance Fields (NeRF), face limitations in synthesizing high-quality novel views when the camera poses of input images are imperfect. To address this issue, we propose a novel 3D reconstruction framework that enables simultaneous optimization of camera poses, dubbed CBARF (Cascaded Bundle-Adjusting NeRF).In a nutshell, our framework optimizes camera poses in a coarse-to-fine manner and then reconstructs scenes based on the rectified poses. It is observed that the initialization of camera poses has a significant impact on the performance of bundle-adjustment (BA). Therefore, we cascade multiple BA modules at different scales to progressively improve the camera poses. Meanwhile, we develop a neighbor-replacement strategy to further optimize the results of BA in each stage. In this step, we introduce a novel criterion to effectively identify poorly estimated camera poses. Then we replace them with the poses of neighboring cameras, thus further eliminating the impact of inaccurate camera poses. Once camera poses have been optimized, we employ a density voxel grid to generate high-quality 3D reconstructed scenes and images in novel views. Experimental results demonstrate that our CBARF model achieves state-of-the-art performance in both pose optimization and novel view synthesis, especially in the existence of large camera pose noise.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The paper aims to address the issue of existing volumetric neural rendering techniques (such as NeRF) struggling to synthesize high-quality novel views when the camera poses of input images are imprecise. To tackle this challenge, the authors propose a new 3D reconstruction framework—Cascaded Bundle Adjustment Neural Radiance Fields (CBARF). Specifically:

- **Objective**: To optimize camera poses, enabling scene reconstruction and novel view generation even when camera poses are noisy or partially unknown.
- **Method**: Gradually optimizing camera poses through a multi-stage (coarse-to-fine) approach and introducing a neighbor replacement strategy to further improve the results of bundle adjustment (BA).
- **Contributions**:
- Proposing a robust coarse-to-fine 3D reconstruction framework that effectively optimizes camera poses under significant noise.
- Discovering that camera pose initialization is crucial for bundle adjustment performance and proposing cascaded bundle adjustment to progressively refine inaccurate camera poses.
- Introducing a neighbor replacement strategy to improve the optimization process by identifying and replacing inaccurate camera poses.

Through experimental validation, CBARF achieves state-of-the-art performance in handling camera pose optimization with noisy or incomplete initial estimates.