Abstract:Recent breakthroughs in Neural Radiance Fields (NeRFs) have sparked significant demand for their integration into real-world 3D applications. However, the varied functionalities required by different 3D applications often necessitate diverse NeRF models with various pipelines, leading to tedious NeRF training for each target task and cumbersome trial-and-error experiments. Drawing inspiration from the generalization capability and adaptability of emerging foundation models, our work aims to develop one general-purpose NeRF for handling diverse 3D tasks. We achieve this by proposing a framework called Omni-Recon, which is capable of (1) generalizable 3D reconstruction and zero-shot multitask scene understanding, and (2) adaptability to diverse downstream 3D applications such as real-time rendering and scene editing. Our key insight is that an image-based rendering pipeline, with accurate geometry and appearance estimation, can lift 2D image features into their 3D counterparts, thus extending widely explored 2D tasks to the 3D world in a generalizable manner. Specifically, our Omni-Recon features a general-purpose NeRF model using image-based rendering with two decoupled branches: one complex transformer-based branch that progressively fuses geometry and appearance features for accurate geometry estimation, and one lightweight branch for predicting blending weights of source views. This design achieves state-of-the-art (SOTA) generalizable 3D surface reconstruction quality with blending weights reusable across diverse tasks for zero-shot multitask scene understanding. In addition, it can enable real-time rendering after baking the complex geometry branch into meshes, swift adaptation to achieve SOTA generalizable 3D understanding performance, and seamless integration with 2D diffusion models for text-guided 3D editing.

ReconFusion: 3D Reconstruction with Diffusion Priors

Deceptive-NeRF/3DGS: Diffusion-Generated Pseudo-Observations for High-Quality Sparse-View Reconstruction

Omni-Recon: Harnessing Image-based Rendering for General-Purpose Neural Radiance Fields

NeRFusion: Fusing Radiance Fields for Large-Scale Scene Reconstruction

Prior-Driven NeRF: Prior Guided Rendering

SparseFusion: Distilling View-conditioned Diffusion for 3D Reconstruction

3D Reconstruction and New View Synthesis of Indoor Environments based on a Dual Neural Radiance Field

NeRF: Neural Radiance Field in 3D Vision, A Comprehensive Review

SN 2 eRF: A Framework for Neural Radiance Fields given Sparse and Noisy Poses

Reconstructive Latent-Space Neural Radiance Fields for Efficient 3D Scene Representations

UNISURF: Unifying Neural Implicit Surfaces and Radiance Fields for Multi-View Reconstruction

$ρ$-NeRF: Leveraging Attenuation Priors in Neural Radiance Field for 3D Computed Tomography Reconstruction

ReconX: Reconstruct Any Scene from Sparse Views with Video Diffusion Model

Dense Depth Priors for Neural Radiance Fields from Sparse Input Views

NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections

Sparse3D: Distilling Multiview-Consistent Diffusion for Object Reconstruction from Sparse Views

PERF: Panoramic Neural Radiance Field From a Single Panorama

An Exploration of Neural Radiance Field Scene Reconstruction: Synthetic, Real-world and Dynamic Scenes

NeRF-US: Removing Ultrasound Imaging Artifacts from Neural Radiance Fields in the Wild

NerfDiff: Single-image View Synthesis with NeRF-guided Distillation from 3D-aware Diffusion

SceneRF: Self-Supervised Monocular 3D Scene Reconstruction with Radiance Fields