Abstract:Visualizing surgical scenes is crucial for revealing internal anatomical structures during minimally invasive procedures. Novel View Synthesis is a vital technique that offers geometry and appearance reconstruction, enhancing understanding, planning, and decision-making in surgical scenes. Despite the impressive achievements of Neural Radiance Field (NeRF), its direct application to surgical scenes produces unsatisfying results due to two challenges: endoscopic sparse views and significant photometric inconsistencies. In this paper, we propose uncertainty-aware conditional NeRF for novel view synthesis to tackle the severe shape-radiance ambiguity from sparse surgical views. The core of UC-NeRF is to incorporate the multi-view uncertainty estimation to condition the neural radiance field for modeling the severe photometric inconsistencies adaptively. Specifically, our UC-NeRF first builds a consistency learner in the form of multi-view stereo network, to establish the geometric correspondence from sparse views and generate uncertainty estimation and feature priors. In neural rendering, we design a base-adaptive NeRF network to exploit the uncertainty estimation for explicitly handling the photometric inconsistencies. Furthermore, an uncertainty-guided geometry distillation is employed to enhance geometry learning. Experiments on the SCARED and Hamlyn datasets demonstrate our superior performance in rendering appearance and geometry, consistently outperforming the current state-of-the-art approaches. Our code will be released at \url{<a class="link-external link-https" href="https://github.com/wrld/UC-NeRF" rel="external noopener nofollow">this https URL</a>}.

Bayes' Rays: Uncertainty Quantification for Neural Radiance Fields

Estimating 3D Uncertainty Field: Quantifying Uncertainty for Neural Radiance Fields

Bayesian NeRF: Quantifying Uncertainty with Volume Density for Neural Implicit Fields

Stochastic Neural Radiance Fields: Quantifying Uncertainty in Implicit 3D Representations

Bayesian uncertainty analysis for underwater 3D reconstruction with neural radiance fields

Neural Visibility Field for Uncertainty-Driven Active Mapping

Density-aware NeRF Ensembles: Quantifying Predictive Uncertainty in Neural Radiance Fields

Uncertainty Quantification of Neural Reflectance Fields for Underwater Scenes

UC-NeRF: Uncertainty-aware Conditional Neural Radiance Fields from Endoscopic Sparse Views

FisherRF: Active View Selection and Uncertainty Quantification for Radiance Fields using Fisher Information

Where and How: Mitigating Confusion in Neural Radiance Fields from Sparse Inputs

Radiance Fields from Photons

Dense Depth Priors for Neural Radiance Fields from Sparse Input Views

Nerf++: Analyzing and improving neural radiance fields

FG-NeRF: Flow-GAN Based Probabilistic Neural Radiance Field for Independence-Assumption-Free Uncertainty Estimation

Taming Uncertainty in Sparse-view Generalizable NeRF via Indirect Diffusion Guidance

SN 2 eRF: A Framework for Neural Radiance Fields given Sparse and Noisy Poses

ActiveNeRF: Learning Where to See with Uncertainty Estimation

UMedNeRF: Uncertainty-aware Single View Volumetric Rendering for Medical Neural Radiance Fields

NeRF Revisited: Fixing Quadrature Instability in Volume Rendering

Area-NeRF: Area-based Neural Radiance Fields