ID-NeRF: Indirect Diffusion-guided Neural Radiance Fields for Generalizable View Synthesis

Yaokun Li,Chao Gou,Guang Tan

2024-01-01

Abstract:Implicit neural representations, represented by Neural Radiance Fields(NeRF), have dominated research in 3D computer vision by virtue of high-qualityvisual results and data-driven benefits. However, their realistic applicationsare hindered by the need for dense inputs and per-scene optimization. To solvethis problem, previous methods implement generalizable NeRFs by extractinglocal features from sparse inputs as conditions for the NeRF decoder. However,although this way can allow feed-forward reconstruction, they suffer from theinherent drawback of yielding sub-optimal results caused by erroneousreprojected features. In this paper, we focus on this problem and aim toaddress it by introducing pre-trained generative priors to enable high-qualitygeneralizable novel view synthesis. Specifically, we propose a novel IndirectDiffusion-guided NeRF framework, termed ID-NeRF, which leverages pre-traineddiffusion priors as a guide for the reprojected features created by theprevious paradigm. Notably, to enable 3D-consistent predictions, the proposedID-NeRF discards the way of direct supervision commonly used in prior 3Dgenerative models and instead adopts a novel indirect prior injection strategy.This strategy is implemented by distilling pre-trained knowledge into animaginative latent space via score-based distillation, and an attention-basedrefinement module is then proposed to leverage the embedded priors to improvereprojected features extracted from sparse inputs. We conduct extensiveexperiments on multiple datasets to evaluate our method, and the resultsdemonstrate the effectiveness of our method in synthesizing novel views in ageneralizable manner, especially in sparse settings.