Abstract:Neural Radiance Field (NeRF) holds immense promise for generating photo-realistic images and videos. How-ever, the computation and memory demands significantly impede its applicability in real-time tasks such as virtual reality and augmented reality. While accelerators dedicatedly designed for NeRF have emerged, they may not be the optimal solution as they only consider the optimization of full-precision NeRF models, leading to unsatisfactory memory and computation saving. This paper proposes Sensitivity-Aware Dynamic Precision Rendering, abbreviated as SRender, an algorithm-hardware co-design framework that borrows the idea of Adaptive Rendering commonly used in conventional graphics rendering to accelerate the rendering process of NeRF exploiting the fine-grained per-ray and per-point levels sensitivity. The fundamental idea behind SRender is that rays and points with high sensitivity require high-precision data types, while the insensitive rays and points do not, allowing them to be sacrificed for greater acceleration. Accordingly, the algorithm part of SRender dynamically configures the precision of the hash table and the multilayer perceptron (MLP), and the hardware part features a specialized accelerator that exploits such mixed-precision memory access and computation patterns. Specifically, we predict the sensitive rays and points based on low-precision hash encoding and feature computation. Then, we trigger a high-precision rendering again on the sensitive ones to guarantee the rendering quality. The hardware design of SRender includes a sensitivity prediction engine, coarse-then-fine-grained encoding engines, and a recovery engine, which translates the computational saving in the algorithm to real speedup. Using the coarse-then-fine-grained encoding engine, we maximize the data reuse on-chip, thereby reducing the stress of off-chip memory bandwidth. Experiments show that the edge-level SRender can achieve 3.94x and 34.06x speedup over state-of-the-art NeRF accelerator NeuRex and Jetson AGX Xavier GPU with negligible rendering quality loss.

Hi-NeRF: A Multicore NeRF Accelerator with Hierarchical Empty Space Skipping for Edge 3-D Rendering

SRender: Boosting Neural Radiance Field Efficiency Via Sensitivity-Aware Dynamic Precision Rendering

RT-NeRF: Real-Time On-Device Neural Radiance Fields Towards Immersive AR/VR Rendering

GNeRF: Accelerating Neural Radiance Fields Inference Via Adaptive Sample Gating

Gen-NeRF: Efficient and Generalizable Neural Radiance Fields via Algorithm-Hardware Co-Design

ICARUS: A Specialized Architecture for Neural Radiance Fields Rendering

ACFNeRF: Accelerating and Cache-Free Neural Rendering Via Point Cloud-Based Distance Fields.

Adaptive Multi-NeRF: Exploit Efficient Parallelism in Adaptive Multiple Scale Neural Radiance Field Rendering

NeRF-PIM: PIM Hardware-Software Co-Design of Neural Rendering Networks

Potamoi: Accelerating Neural Rendering via a Unified Streaming Architecture

NeRF-Learner: A 2.79mj/frame NeRF-SLAM Processor with Unified Inference/Training Compute-in-Memory for Large-Scale Neural Rendering

MixRT: Mixed Neural Representations For Real-Time NeRF Rendering

Hyb-NeRF: A Multiresolution Hybrid Encoding for Neural Radiance Fields

HeRF: A Hierarchical Framework for Efficient and Extendable New View Synthesis

HollowNeRF: Pruning Hashgrid-Based NeRFs with Trainable Collision Mitigation

SteerNeRF: Accelerating NeRF Rendering via Smooth Viewpoint Trajectory

FastNeRF: High-Fidelity Neural Rendering at 200FPS

A General Implicit Framework for Fast NeRF Composition and Rendering

Cicero: Addressing Algorithmic and Architectural Bottlenecks in Neural Rendering by Radiance Warping and Memory Optimizations

DONeRF: Towards Real‐Time Rendering of Compact Neural Radiance Fields using Depth Oracle Networks

Enhance-NeRF: Multiple Performance Evaluation for Neural Radiance Fields