Abstract:Although Neural Radiance Fields (NeRFs) have markedly improved novel view synthesis, accurate uncertainty quantification in their image predictions remains an open problem. The prevailing methods for estimating uncertainty, including the state-of-the-art Density-aware NeRF Ensembles (DANE) [29], quantify uncertainty without calibration. This frequently leads to over- or under-confidence in image predictions, which can undermine their real-world applications. In this paper, we propose a method which, for the first time, achieves calibrated uncertainties for NeRFs. To accomplish this, we overcome a significant challenge in adapting existing calibration techniques to NeRFs: a need to hold out ground truth images from the target scene, reducing the number of images left to train the NeRF. This issue is particularly problematic in sparse-view settings, where we can operate with as few as three images. To address this, we introduce the concept of a meta-calibrator that performs uncertainty calibration for NeRFs with a single forward pass without the need for holding out any images from the target scene. Our meta-calibrator is a neural network that takes as input the NeRF images and uncalibrated uncertainty maps and outputs a scene-specific calibration curve that corrects the NeRF's uncalibrated uncertainties. We show that the meta-calibrator can generalize on unseen scenes and achieves well-calibrated and state-of-the-art uncertainty for NeRFs, significantly beating DANE and other approaches. This opens opportunities to improve applications that rely on accurate NeRF uncertainty estimates such as next-best view planning and potentially more trustworthy image reconstruction for medical diagnosis. The code is available at <a class="link-external link-https" href="https://niki-amini-naieni.github.io/instantcalibration.github.io/" rel="external noopener nofollow">this https URL</a>.

What problem does this paper attempt to address?

The problem this paper attempts to address is how to achieve accurate uncertainty quantification in the prediction of new viewpoint images generated by Neural Radiance Fields (NeRFs). Although existing methods such as Density-Aware NeRF Ensemble (DANE) can estimate uncertainty, these estimates lack calibration, leading to overconfidence or underconfidence in image predictions in practical applications. This limits the use of NeRFs in applications requiring high reliability, such as MRI image reconstruction based on sparse data or next-best-view planning. To solve this problem, the paper proposes a new method—the meta-calibrator, which for the first time enables calibrated uncertainty estimation for NeRFs. The meta-calibrator can complete uncertainty calibration with a single forward pass, without reserving any real images from the target scene. This overcomes a major challenge of applying existing calibration techniques to NeRFs, where reserving images reduces the number of images available for training NeRF, especially in sparse viewpoint settings with only a few images, where this impact is particularly significant. Specifically, the meta-calibrator is a neural network that takes as input the images generated by NeRF and the uncalibrated uncertainty maps, and outputs a scene-specific calibration curve to correct the uncalibrated uncertainty of NeRF. Experimental results show that the meta-calibrator can generalize to unseen scenes and achieve more accurate and advanced uncertainty estimation than DANE and other methods. This is significant for applications that rely on accurate NeRF uncertainty estimation, such as next-best-view planning and potentially more reliable medical diagnostic image reconstruction.

Instant Uncertainty Calibration of NeRFs Using a Meta-Calibrator

UC-NeRF: Uncertainty-aware Conditional Neural Radiance Fields from Endoscopic Sparse Views

Density-aware NeRF Ensembles: Quantifying Predictive Uncertainty in Neural Radiance Fields

NeRF On-the-go: Exploiting Uncertainty for Distractor-free NeRFs in the Wild

ActiveNeRF: Learning Where to See with Uncertainty Estimation

Estimating 3D Uncertainty Field: Quantifying Uncertainty for Neural Radiance Fields

Neural Visibility Field for Uncertainty-Driven Active Mapping

Stochastic Neural Radiance Fields: Quantifying Uncertainty in Implicit 3D Representations

Taming Uncertainty in Sparse-view Generalizable NeRF via Indirect Diffusion Guidance

Towards calibrated and scalable uncertainty representations for neural networks

Density Uncertainty Quantification with NeRF-Ensembles: Impact of Data and Scene Constraints

Depth-guided NeRF Training via Earth Mover's Distance

NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections

Self-NeRF: A Self-Training Pipeline for Few-Shot Neural Radiance Fields

Bayes' Rays: Uncertainty Quantification for Neural Radiance Fields

SCADE: NeRFs from Space Carving with Ambiguity-Aware Depth Estimates

Uncertainty modeling for fine-tuned implicit functions

DaRF: Boosting Radiance Fields from Sparse Inputs with Monocular Depth Adaptation

IOVS4NeRF:Incremental Optimal View Selection for Large-Scale NeRFs

ViewNeRF: Unsupervised Viewpoint Estimation Using Category-Level Neural Radiance Fields

HyperNeRF: A Higher-Dimensional Representation for Topologically Varying Neural Radiance Fields