Abstract:Neural Radiance Field (NeRF) holds immense promise for generating photo-realistic images and videos. How-ever, the computation and memory demands significantly impede its applicability in real-time tasks such as virtual reality and augmented reality. While accelerators dedicatedly designed for NeRF have emerged, they may not be the optimal solution as they only consider the optimization of full-precision NeRF models, leading to unsatisfactory memory and computation saving. This paper proposes Sensitivity-Aware Dynamic Precision Rendering, abbreviated as SRender, an algorithm-hardware co-design framework that borrows the idea of Adaptive Rendering commonly used in conventional graphics rendering to accelerate the rendering process of NeRF exploiting the fine-grained per-ray and per-point levels sensitivity. The fundamental idea behind SRender is that rays and points with high sensitivity require high-precision data types, while the insensitive rays and points do not, allowing them to be sacrificed for greater acceleration. Accordingly, the algorithm part of SRender dynamically configures the precision of the hash table and the multilayer perceptron (MLP), and the hardware part features a specialized accelerator that exploits such mixed-precision memory access and computation patterns. Specifically, we predict the sensitive rays and points based on low-precision hash encoding and feature computation. Then, we trigger a high-precision rendering again on the sensitive ones to guarantee the rendering quality. The hardware design of SRender includes a sensitivity prediction engine, coarse-then-fine-grained encoding engines, and a recovery engine, which translates the computational saving in the algorithm to real speedup. Using the coarse-then-fine-grained encoding engine, we maximize the data reuse on-chip, thereby reducing the stress of off-chip memory bandwidth. Experiments show that the edge-level SRender can achieve 3.94x and 34.06x speedup over state-of-the-art NeRF accelerator NeuRex and Jetson AGX Xavier GPU with negligible rendering quality loss.

Potamoi: Accelerating Neural Rendering via a Unified Streaming Architecture

RT-NeRF: Real-Time On-Device Neural Radiance Fields Towards Immersive AR/VR Rendering

SRender: Boosting Neural Radiance Field Efficiency Via Sensitivity-Aware Dynamic Precision Rendering

Adaptive Multi-NeRF: Exploit Efficient Parallelism in Adaptive Multiple Scale Neural Radiance Field Rendering

Cicero: Addressing Algorithmic and Architectural Bottlenecks in Neural Rendering by Radiance Warping and Memory Optimizations

ACFNeRF: Accelerating and Cache-Free Neural Rendering Via Point Cloud-Based Distance Fields.

ICARUS: A Specialized Architecture for Neural Radiance Fields Rendering

NeRF-PIM: PIM Hardware-Software Co-Design of Neural Rendering Networks

Hi-NeRF: A Multicore NeRF Accelerator with Hierarchical Empty Space Skipping for Edge 3-D Rendering

SteerNeRF: Accelerating NeRF Rendering via Smooth Viewpoint Trajectory

MixRT: Mixed Neural Representations For Real-Time NeRF Rendering

Gen-NeRF: Efficient and Generalizable Neural Radiance Fields via Algorithm-Hardware Co-Design

GNeRF: Accelerating Neural Radiance Fields Inference Via Adaptive Sample Gating

FastNeRF: High-Fidelity Neural Rendering at 200FPS

SpikingNeRF: Making Bio-inspired Neural Networks See through the Real World

NeRFBuff: Fast Neural Rendering via Inter-frame Feature Buffering

DONeRF: Towards Real‐Time Rendering of Compact Neural Radiance Fields using Depth Oracle Networks

AdaNeRF: Adaptive Sampling for Real-time Rendering of Neural Radiance Fields

NeRF-Learner: A 2.79mj/frame NeRF-SLAM Processor with Unified Inference/Training Compute-in-Memory for Large-Scale Neural Rendering

Polynomial for real-time rendering of neural radiance fields

NEPHELE: A Neural Platform for Highly Realistic Cloud Radiance Rendering