Abstract:Recent breakthroughs in Neural Radiance Fields (NeRFs) have sparked significant demand for their integration into real-world 3D applications. However, the varied functionalities required by different 3D applications often necessitate diverse NeRF models with various pipelines, leading to tedious NeRF training for each target task and cumbersome trial-and-error experiments. Drawing inspiration from the generalization capability and adaptability of emerging foundation models, our work aims to develop one general-purpose NeRF for handling diverse 3D tasks. We achieve this by proposing a framework called Omni-Recon, which is capable of (1) generalizable 3D reconstruction and zero-shot multitask scene understanding, and (2) adaptability to diverse downstream 3D applications such as real-time rendering and scene editing. Our key insight is that an image-based rendering pipeline, with accurate geometry and appearance estimation, can lift 2D image features into their 3D counterparts, thus extending widely explored 2D tasks to the 3D world in a generalizable manner. Specifically, our Omni-Recon features a general-purpose NeRF model using image-based rendering with two decoupled branches: one complex transformer-based branch that progressively fuses geometry and appearance features for accurate geometry estimation, and one lightweight branch for predicting blending weights of source views. This design achieves state-of-the-art (SOTA) generalizable 3D surface reconstruction quality with blending weights reusable across diverse tasks for zero-shot multitask scene understanding. In addition, it can enable real-time rendering after baking the complex geometry branch into meshes, swift adaptation to achieve SOTA generalizable 3D understanding performance, and seamless integration with 2D diffusion models for text-guided 3D editing.

MVSNeRF: Fast Generalizable Radiance Field Reconstruction from Multi-View Stereo

MPS-NeRF: Generalizable 3D Human Rendering from Multiview Images

SG-NeRF: Sparse-Input Generalized Neural Radiance Fields for Novel View Synthesis

Multi-View Stereo and Depth Priors Guided NeRF for View Synthesis.

MMNeRF: Multi-Modal and Multi-View Optimized Cross-Scene Neural Radiance Fields

NeRFusion: Fusing Radiance Fields for Large-Scale Scene Reconstruction

Cascaded and Generalizable Neural Radiance Fields for Fast View Synthesis

BoostMVSNeRFs: Boosting MVS-based NeRFs to Generalizable View Synthesis in Large-scale Scenes

Neural Radiance Fields Approach to Deep Multi-View Photometric Stereo

Fast Radiance Field Reconstruction from Sparse Inputs

Generalizable Neural Radiance Field with Hierarchical Geometry Constraint.

Multi-view stereo-regulated NeRF for urban scene novel view synthesis

NeRF-SDP: Efficient Generalizable Neural Radiance Field with Scene Depth Perception.

Omni-Recon: Harnessing Image-based Rendering for General-Purpose Neural Radiance Fields

TransNeRF: Multi-View Optimization for General Neural Radiance Fields Across Scenes

Volume Feature Rendering for Fast Neural Radiance Field Reconstruction

MM-NeRF: Large-Scale Scene Representation with Multi-Resolution Hash Grid and Multi-View Priors Features

GM-NeRF: Learning Generalizable Model-based Neural Radiance Fields from Multi-view Images

3D Reconstruction and Rendering Based on Improved Neural Radiance Field

Depth-Guided Optimization of Neural Radiance Fields for Indoor Multi-View Stereo

Single-view Neural Radiance Fields with Depth Teacher