Abstract:Neural Radiance Field (NeRF) holds immense promise for generating photo-realistic images and videos. How-ever, the computation and memory demands significantly impede its applicability in real-time tasks such as virtual reality and augmented reality. While accelerators dedicatedly designed for NeRF have emerged, they may not be the optimal solution as they only consider the optimization of full-precision NeRF models, leading to unsatisfactory memory and computation saving. This paper proposes Sensitivity-Aware Dynamic Precision Rendering, abbreviated as SRender, an algorithm-hardware co-design framework that borrows the idea of Adaptive Rendering commonly used in conventional graphics rendering to accelerate the rendering process of NeRF exploiting the fine-grained per-ray and per-point levels sensitivity. The fundamental idea behind SRender is that rays and points with high sensitivity require high-precision data types, while the insensitive rays and points do not, allowing them to be sacrificed for greater acceleration. Accordingly, the algorithm part of SRender dynamically configures the precision of the hash table and the multilayer perceptron (MLP), and the hardware part features a specialized accelerator that exploits such mixed-precision memory access and computation patterns. Specifically, we predict the sensitive rays and points based on low-precision hash encoding and feature computation. Then, we trigger a high-precision rendering again on the sensitive ones to guarantee the rendering quality. The hardware design of SRender includes a sensitivity prediction engine, coarse-then-fine-grained encoding engines, and a recovery engine, which translates the computational saving in the algorithm to real speedup. Using the coarse-then-fine-grained encoding engine, we maximize the data reuse on-chip, thereby reducing the stress of off-chip memory bandwidth. Experiments show that the edge-level SRender can achieve 3.94x and 34.06x speedup over state-of-the-art NeRF accelerator NeuRex and Jetson AGX Xavier GPU with negligible rendering quality loss.

GNeRF: Accelerating Neural Radiance Fields Inference Via Adaptive Sample Gating

Adaptive Multi-NeRF: Exploit Efficient Parallelism in Adaptive Multiple Scale Neural Radiance Field Rendering

Gen-NeRF: Efficient and Generalizable Neural Radiance Fields via Algorithm-Hardware Co-Design

FastNeRF: High-Fidelity Neural Rendering at 200FPS

ACFNeRF: Accelerating and Cache-Free Neural Rendering Via Point Cloud-Based Distance Fields.

SRender: Boosting Neural Radiance Field Efficiency Via Sensitivity-Aware Dynamic Precision Rendering

KiloNeRF: Speeding Up Neural Radiance Fields with Thousands of Tiny MLPs

Efficient NeRF Optimization -- Not All Samples Remain Equally Hard

RT-NeRF: Real-Time On-Device Neural Radiance Fields Towards Immersive AR/VR Rendering

ICARUS: A Specialized Architecture for Neural Radiance Fields Rendering

Hi-NeRF: A Multicore NeRF Accelerator with Hierarchical Empty Space Skipping for Edge 3-D Rendering

GL-NeRF: Gauss-Laguerre Quadrature Enables Training-Free NeRF Acceleration

PyNeRF: Pyramidal Neural Radiance Fields

Potamoi: Accelerating Neural Rendering via a Unified Streaming Architecture

SGCNeRF: Few-Shot Neural Rendering via Sparse Geometric Consistency Guidance

Hyb-NeRF: A Multiresolution Hybrid Encoding for Neural Radiance Fields

Baking Neural Radiance Fields for Real-Time View Synthesis

A General Implicit Framework for Fast NeRF Composition and Rendering

Analyzing the Internals of Neural Radiance Fields

DONeRF: Towards Real‐Time Rendering of Compact Neural Radiance Fields using Depth Oracle Networks

GANeRF: Leveraging Discriminators to Optimize Neural Radiance Fields