Abstract:Neural Radiance Fields (NeRF) have demonstrated impressive potential in synthesizing novel views from dense input, however, their effectiveness is challenged when dealing with sparse input. Existing approaches that incorporate additional depth or semantic supervision can alleviate this issue to an extent. However, the process of supervision collection is not only costly but also potentially inaccurate, leading to poor performance and generalization ability in diverse scenarios. In our work, we introduce a novel model: the Collaborative Neural Radiance Fields (ColNeRF) designed to work with sparse input. The collaboration in ColNeRF includes both the cooperation between sparse input images and the cooperation between the output of the neural radiation field. Through this, we construct a novel collaborative module that aligns information from various views and meanwhile imposes self-supervised constraints to ensure multi-view consistency in both geometry and appearance. A Collaborative Cross-View Volume Integration module (CCVI) is proposed to capture complex occlusions and implicitly infer the spatial location of objects. Moreover, we introduce self-supervision of target rays projected in multiple directions to ensure geometric and color consistency in adjacent regions. Benefiting from the collaboration at the input and output ends, ColNeRF is capable of capturing richer and more generalized scene representation, thereby facilitating higher-quality results of the novel view synthesis. Extensive experiments demonstrate that ColNeRF outperforms state-of-the-art sparse input generalizable NeRF methods. Furthermore, our approach exhibits superiority in fine-tuning towards adapting to new scenes, achieving competitive performance compared to per-scene optimized NeRF-based methods while significantly reducing computational costs. Our code is available at: <a class="link-external link-https" href="https://github.com/eezkni/ColNeRF" rel="external noopener nofollow">this https URL</a>.

Structure-aware neural radiance fields without posed camera

MPS-NeRF: Generalizable 3D Human Rendering from Multiview Images

SN 2 eRF: A Framework for Neural Radiance Fields given Sparse and Noisy Poses

StructNeRF: Neural Radiance Fields for Indoor Scenes With Structural Hints

Collaborative neural radiance fields for novel view synthesis

SPARF: Neural Radiance Fields from Sparse and Noisy Poses

Pose-Free Neural Radiance Fields via Implicit Pose Regularization

Dense Depth Priors for Neural Radiance Fields from Sparse Input Views

Enhancing Neural Radiance Fields with Depth and Normal Completion Priors from Sparse Views

FDC-NeRF: Learning Pose-Free Neural Radiance Fields with Flow-Depth Consistency.

CT-NeRF: Incremental Optimizing Neural Radiance Field and Poses with Complex Trajectory

CBARF: Cascaded Bundle-Adjusting Neural Radiance Fields from Imperfect Camera Poses

NeFF: Neural Feature Fields for Few-Shot View Synthesis Addressing the Shape-Radiance Ambiguity

Novel View Synthesis with Neural Radiance Fields for Industrial Robot Applications

GM-NeRF: Learning Generalizable Model-based Neural Radiance Fields from Multi-view Images

Neural Radiance Fields From Sparse RGB-D Images for High-Quality View Synthesis

MC-NeRF: Multi-Camera Neural Radiance Fields for Multi-Camera Image Acquisition Systems

SG-NeRF: Sparse-Input Generalized Neural Radiance Fields for Novel View Synthesis

TD-NeRF: Novel Truncated Depth Prior for Joint Camera Pose and Neural Radiance Field Optimization

ColNeRF: Collaboration for Generalizable Sparse Input Neural Radiance Field

BeNeRF: Neural Radiance Fields from a Single Blurry Image and Event Stream