Abstract:Neural Radiance Fields (NeRF) have demonstrated impressive potential in synthesizing novel views from dense input, however, their effectiveness is challenged when dealing with sparse input. Existing approaches that incorporate additional depth or semantic supervision can alleviate this issue to an extent. However, the process of supervision collection is not only costly but also potentially inaccurate, leading to poor performance and generalization ability in diverse scenarios. In our work, we introduce a novel model: the Collaborative Neural Radiance Fields (ColNeRF) designed to work with sparse input. The collaboration in ColNeRF includes both the cooperation between sparse input images and the cooperation between the output of the neural radiation field. Through this, we construct a novel collaborative module that aligns information from various views and meanwhile imposes self-supervised constraints to ensure multi-view consistency in both geometry and appearance. A Collaborative Cross-View Volume Integration module (CCVI) is proposed to capture complex occlusions and implicitly infer the spatial location of objects. Moreover, we introduce self-supervision of target rays projected in multiple directions to ensure geometric and color consistency in adjacent regions. Benefiting from the collaboration at the input and output ends, ColNeRF is capable of capturing richer and more generalized scene representation, thereby facilitating higher-quality results of the novel view synthesis. Extensive experiments demonstrate that ColNeRF outperforms state-of-the-art sparse input generalizable NeRF methods. Furthermore, our approach exhibits superiority in fine-tuning towards adapting to new scenes, achieving competitive performance compared to per-scene optimized NeRF-based methods while significantly reducing computational costs. Our code is available at: <a class="link-external link-https" href="https://github.com/eezkni/ColNeRF" rel="external noopener nofollow">this https URL</a>.

UC-NeRF: Uncertainty-aware Conditional Neural Radiance Fields from Endoscopic Sparse Views

UMedNeRF: Uncertainty-aware Single View Volumetric Rendering for Medical Neural Radiance Fields

SSNeRF: Sparse View Semi-supervised Neural Radiance Fields with Augmentation

Collaborative neural radiance fields for novel view synthesis

Improving Neural Radiance Fields with Depth-aware Optimization for Novel View Synthesis

Enhancing Neural Radiance Fields with Depth and Normal Completion Priors from Sparse Views

SN 2 eRF: A Framework for Neural Radiance Fields given Sparse and Noisy Poses

IOVS4NeRF:Incremental Optimal View Selection for Large-Scale NeRFs

Neural radiance fields-based multi-view endoscopic scene reconstruction for surgical simulation

UC-NeRF: Neural Radiance Field for Under-Calibrated Multi-view Cameras in Autonomous Driving

Where and How: Mitigating Confusion in Neural Radiance Fields from Sparse Inputs

Multi-View Stereo and Depth Priors Guided NeRF for View Synthesis.

RegNeRF: Regularizing Neural Radiance Fields for View Synthesis from Sparse Inputs

ActiveNeRF: Learning Where to See with Uncertainty Estimation

HG3-NeRF: Hierarchical Geometric, Semantic, and Photometric Guided Neural Radiance Fields for Sparse View Inputs

Neural Radiance Fields for Novel View Synthesis in Monocular Gastroscopy

NeRFVS: Neural Radiance Fields for Free View Synthesis Via Geometry Scaffolds

Estimating 3D Uncertainty Field: Quantifying Uncertainty for Neural Radiance Fields

SGCNeRF: Few-Shot Neural Rendering via Sparse Geometric Consistency Guidance

ColNeRF: Collaboration for Generalizable Sparse Input Neural Radiance Field

Learning Geometry Consistent Neural Radiance Fields from Sparse and Unposed Views