Abstract:Recent advances in neural rendering have shown that although computationally expensive and slow for training, implicit compact models can accurately learn a scene's geometries and view-dependent appearances from multiple views. To maintain such a small memory footprint but achieve faster inference times, recent works have adopted 'sampler' networks that adaptively sample a small subset of points along each ray in the implicit neural radiance fields (NeRF), effectively reducing the number of network forward passes to render a ray color. Although these methods achieve up to a reduction in rendering time, they still suffer from considerable quality degradation compared to vanilla NeRF. In contrast, we propose a new projection-aware neural radiance field model, referred to as ProNeRF, which provides an optimal trade-off between the memory footprint (similar to NeRF), speed (faster than HyperReel), and quality (better than -Planes). ProNeRF is equipped with a novel projection-aware sampling (PAS) network together with a new training strategy for ray exploration and exploitation, allowing for efficient fine-grained particle sampling. Our exploration and exploitation training strategy allows ProNeRF to learn the color and density distributions of full scenes, while also learning efficient ray sampling focused on the highest-density regions. ProNeRF yields state-of-the-art metrics, being 15 to faster with 0.65dB higher PSNR than the vanilla NeRF and showing 0.95dB higher PSNR performance compared to the best published sampler-based method, HyperReel. We provide extensive experimental results that support the effectiveness of our method on the widely adopted forward-facing and 360 datasets, LLFF and Blender, respectively. Additionally, we present real-world applications of ProNeRF on hand-held captured scenes. Our project page is publicly available at https://kaist-viclab.github.io/pronerf-site.

PMPI: Patch-Based Multiplane Images for Real-Time Rendering of Neural Radiance Fields.

MMPI: a Flexible Radiance Field Representation by Multiple Multi-plane Images Blending

Multi-Plane Neural Radiance Fields for Novel View Synthesis

rpcPRF: Generalizable MPI Neural Radiance Field for Satellite Camera with Single and Sparse Views

Tri-MipRF: Tri-Mip Representation for Efficient Anti-Aliasing Neural Radiance Fields

Polynomial for real-time rendering of neural radiance fields

Learning Neural Duplex Radiance Fields for Real-Time View Synthesis

Dense Viewpoint Encoding of 3D Light Fields Based on Neural Graphics Primitives

INPC: Implicit Neural Point Clouds for Radiance Field Rendering

Neural Radiance Fields From Sparse RGB-D Images for High-Quality View Synthesis

Baking Neural Radiance Fields for Real-Time View Synthesis

ProNeRF: Learning Efficient Projection-Aware Ray Sampling for Fine-Grained Implicit Neural Radiance Fields

Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields

VPRF: Visual Perceptual Radiance Fields for Foveated Image Synthesis

Deep Real-time Volumetric Rendering Using Multi-feature Fusion.

Multiscale Representation for Real-Time Anti-Aliasing Neural Rendering

MMNeRF: Multi-Modal and Multi-View Optimized Cross-Scene Neural Radiance Fields

Neural Relightable Participating Media Rendering

Radiance Fields from Photons

Multiplane Prior Guided Few-Shot Aerial Scene Rendering