Abstract:<p>The recent research explosion around implicit neural representations, such as NeRF, shows that there is immense potential for implicitly storing high-quality scene and lighting information in compact neural networks. However, one major limitation preventing the use of NeRF in real-time rendering applications is the prohibitive computational cost of excessive network evaluations along each view ray, requiring dozens of petaFLOPS. In this work, we bring compact neural representations closer to practical rendering of synthetic content in real-time applications, such as games and virtual reality. We show that the number of samples required for each view ray can be significantly reduced when samples are placed around surfaces in the scene without compromising image quality. To this end, we propose a depth oracle network that predicts ray sample locations for each view ray with a single network evaluation. We show that using a classification network around logarithmically discretized and spherically warped depth values is essential to encode surface locations rather than directly estimating depth. The combination of these techniques leads to DONeRF, our compact dual network design with a depth oracle network as its first step and a locally sampled shading network for ray accumulation. With DONeRF, we reduce the inference costs by up to 48× compared to NeRF when conditioning on available ground truth depth information. Compared to concurrent acceleration methods for raymarching-based neural representations, DONeRF does not require additional memory for explicit caching or acceleration structures, and can render interactively (20 frames per second) on a single GPU.</p>

FPO++: efficient encoding and rendering of dynamic neural radiance fields by analyzing and enhancing Fourier PlenOctrees

FPO++: Efficient Encoding and Rendering of Dynamic Neural Radiance Fields by Analyzing and Enhancing Fourier PlenOctrees

Fourier PlenOctrees for Dynamic Radiance Field Rendering in Real-time

PlenOctrees for Real-time Rendering of Neural Radiance Fields

Dynamic PlenOctree for Adaptive Sampling Refinement in Explicit NeRF

Polynomial for real-time rendering of neural radiance fields

Tetra-NeRF: Representing Neural Radiance Fields Using Tetrahedra

Compact Real-time Radiance Fields with Neural Codebook

Plenoptic PNG: Real-Time Neural Radiance Fields in 150 KB

Adaptive Shells for Efficient Neural Radiance Field Rendering

DONeRF: Towards Real‐Time Rendering of Compact Neural Radiance Fields using Depth Oracle Networks

Ced-NeRF: A Compact and Efficient Method for Dynamic Neural Radiance Fields

Neural NeRF Compression

Hyb-NeRF: A Multiresolution Hybrid Encoding for Neural Radiance Fields

RT-Octree: Accelerate PlenOctree Rendering with Batched Regular Tracking and Neural Denoising for Real-time Neural Radiance Fields.

ACFNeRF: Accelerating and Cache-Free Neural Rendering Via Point Cloud-Based Distance Fields.

AdaNeRF: Adaptive Sampling for Real-time Rendering of Neural Radiance Fields

Adaptive Multi-NeRF: Exploit Efficient Parallelism in Adaptive Multiple Scale Neural Radiance Field Rendering

HybridNeRF: Efficient Neural Rendering via Adaptive Volumetric Surfaces

Nerf++: Analyzing and improving neural radiance fields

Reconstructive Latent-Space Neural Radiance Fields for Efficient 3D Scene Representations