The NeRFect Match: Exploring NeRF Features for Visual Localization

Qunjie Zhou,Maxim Maximov,Or Litany,Laura Leal-Taixé

2024-08-21

Abstract:In this work, we propose the use of Neural Radiance Fields (NeRF) as a scene representation for visual localization. Recently, NeRF has been employed to enhance pose regression and scene coordinate regression models by augmenting the training database, providing auxiliary supervision through rendered images, or serving as an iterative refinement module. We extend its recognized advantages -- its ability to provide a compact scene representation with realistic appearances and accurate geometry -- by exploring the potential of NeRF's internal features in establishing precise 2D-3D matches for localization. To this end, we conduct a comprehensive examination of NeRF's implicit knowledge, acquired through view synthesis, for matching under various conditions. This includes exploring different matching network architectures, extracting encoder features at multiple layers, and varying training configurations. Significantly, we introduce NeRFMatch, an advanced 2D-3D matching function that capitalizes on the internal knowledge of NeRF learned via view synthesis. Our evaluation of NeRFMatch on standard localization benchmarks, within a structure-based pipeline, sets a new state-of-the-art for localization performance on Cambridge Landmarks.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The main goal of this paper is to explore the potential application of internal features of Neural Radiance Fields (NeRF) in visual localization tasks. Specifically:

1. **Propose NeRFMatch**: The authors introduce an advanced 2D-3D matching function called NeRFMatch, which leverages the internal knowledge learned by NeRF through view synthesis. NeRFMatch aims to establish precise 2D-3D correspondences between query images and NeRF scene points.

2. **Comprehensive evaluation of NeRF internal features**: By studying different matching network architectures, multiple layers of extracted encoder features, and variations in training configurations, the authors conduct a comprehensive evaluation of NeRF's implicit knowledge.

3. **Enhance localization accuracy**: The research demonstrates how to utilize NeRF's internal features to improve the accuracy of visual localization and proposes two iterative methods for refining pose estimation.

4. **Experimental validation**: The authors evaluate the performance of NeRFMatch on standard localization benchmark datasets (such as Cambridge Landmarks) and achieve competitive results.

In summary, this paper aims to demonstrate that NeRF not only provides realistic appearance and accurate geometric representation but also that its internal features can be used for efficient 2D-3D matching, thereby achieving high-precision visual localization.