Abstract:Implicit neural representations, represented by Neural Radiance Fields(NeRF), have dominated research in 3D computer vision by virtue of high-qualityvisual results and data-driven benefits. However, their realistic applicationsare hindered by the need for dense inputs and per-scene optimization. To solvethis problem, previous methods implement generalizable NeRFs by extractinglocal features from sparse inputs as conditions for the NeRF decoder. However,although this way can allow feed-forward reconstruction, they suffer from theinherent drawback of yielding sub-optimal results caused by erroneousreprojected features. In this paper, we focus on this problem and aim toaddress it by introducing pre-trained generative priors to enable high-qualitygeneralizable novel view synthesis. Specifically, we propose a novel IndirectDiffusion-guided NeRF framework, termed ID-NeRF, which leverages pre-traineddiffusion priors as a guide for the reprojected features created by theprevious paradigm. Notably, to enable 3D-consistent predictions, the proposedID-NeRF discards the way of direct supervision commonly used in prior 3Dgenerative models and instead adopts a novel indirect prior injection strategy.This strategy is implemented by distilling pre-trained knowledge into animaginative latent space via score-based distillation, and an attention-basedrefinement module is then proposed to leverage the embedded priors to improvereprojected features extracted from sparse inputs. We conduct extensiveexperiments on multiple datasets to evaluate our method, and the resultsdemonstrate the effectiveness of our method in synthesizing novel views in ageneralizable manner, especially in sparse settings.

Learning a Diffusion Prior for NeRFs

DiffusioNeRF: Regularizing Neural Radiance Fields with Denoising Diffusion Models

ID-NeRF: Indirect Diffusion-guided Neural Radiance Fields for Generalizable View Synthesis

Taming Uncertainty in Sparse-view Generalizable NeRF via Indirect Diffusion Guidance

Single-Stage Diffusion NeRF: A Unified Approach to 3D Generation and Reconstruction.

Taming Latent Diffusion Model for Neural Radiance Field Inpainting

Prior-Driven NeRF: Prior Guided Rendering

Learn to Optimize Denoising Scores for 3D Generation: A Unified and Improved Diffusion Prior on NeRF and 3D Gaussian Splatting

Deceptive-NeRF/3DGS: Diffusion-Generated Pseudo-Observations for High-Quality Sparse-View Reconstruction

RegNeRF: Regularizing Neural Radiance Fields for View Synthesis from Sparse Inputs

NeRF: Neural Radiance Field in 3D Vision, A Comprehensive Review

SN 2 eRF: A Framework for Neural Radiance Fields given Sparse and Noisy Poses

Dense Depth Priors for Neural Radiance Fields from Sparse Input Views

Federated Neural Radiance Fields

CorresNeRF: Image Correspondence Priors for Neural Radiance Fields

Instant Continual Learning of Neural Radiance Fields

Inpaint4DNeRF: Promptable Spatio-Temporal NeRF Inpainting with Generative Diffusion Models

ReconFusion: 3D Reconstruction with Diffusion Priors

ActiveNeRF: Learning Where to See with Uncertainty Estimation

Enhancing Neural Radiance Fields with Depth and Normal Completion Priors from Sparse Views

Depth-guided NeRF Training via Earth Mover's Distance