Abstract:Neural Radiance Field (NeRF) holds immense promise for generating photo-realistic images and videos. How-ever, the computation and memory demands significantly impede its applicability in real-time tasks such as virtual reality and augmented reality. While accelerators dedicatedly designed for NeRF have emerged, they may not be the optimal solution as they only consider the optimization of full-precision NeRF models, leading to unsatisfactory memory and computation saving. This paper proposes Sensitivity-Aware Dynamic Precision Rendering, abbreviated as SRender, an algorithm-hardware co-design framework that borrows the idea of Adaptive Rendering commonly used in conventional graphics rendering to accelerate the rendering process of NeRF exploiting the fine-grained per-ray and per-point levels sensitivity. The fundamental idea behind SRender is that rays and points with high sensitivity require high-precision data types, while the insensitive rays and points do not, allowing them to be sacrificed for greater acceleration. Accordingly, the algorithm part of SRender dynamically configures the precision of the hash table and the multilayer perceptron (MLP), and the hardware part features a specialized accelerator that exploits such mixed-precision memory access and computation patterns. Specifically, we predict the sensitive rays and points based on low-precision hash encoding and feature computation. Then, we trigger a high-precision rendering again on the sensitive ones to guarantee the rendering quality. The hardware design of SRender includes a sensitivity prediction engine, coarse-then-fine-grained encoding engines, and a recovery engine, which translates the computational saving in the algorithm to real speedup. Using the coarse-then-fine-grained encoding engine, we maximize the data reuse on-chip, thereby reducing the stress of off-chip memory bandwidth. Experiments show that the edge-level SRender can achieve 3.94x and 34.06x speedup over state-of-the-art NeRF accelerator NeuRex and Jetson AGX Xavier GPU with negligible rendering quality loss.

NeRF-PIM: PIM Hardware-Software Co-Design of Neural Rendering Networks

Neural-PIM: Efficient Processing-In-Memory with Neural Approximation of Peripherals

RT-NeRF: Real-Time On-Device Neural Radiance Fields Towards Immersive AR/VR Rendering

Potamoi: Accelerating Neural Rendering via a Unified Streaming Architecture

Hi-NeRF: A Multicore NeRF Accelerator with Hierarchical Empty Space Skipping for Edge 3-D Rendering

Cicero: Addressing Algorithmic and Architectural Bottlenecks in Neural Rendering by Radiance Warping and Memory Optimizations

ICARUS: A Specialized Architecture for Neural Radiance Fields Rendering

SRender: Boosting Neural Radiance Field Efficiency Via Sensitivity-Aware Dynamic Precision Rendering

GNeRF: Accelerating Neural Radiance Fields Inference Via Adaptive Sample Gating

Hyb-NeRF: A Multiresolution Hybrid Encoding for Neural Radiance Fields

Reliability-Aware Training and Performance Modeling for Processing-In-Memory Systems

MF-NeRF: Memory Efficient NeRF with Mixed-Feature Hash Table

Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields

Gen-NeRF: Efficient and Generalizable Neural Radiance Fields via Algorithm-Hardware Co-Design

Enhance-NeRF: Multiple Performance Evaluation for Neural Radiance Fields

NeuPIMs: NPU-PIM Heterogeneous Acceleration for Batched LLM Inferencing

MemANNS: Enhancing Billion-Scale ANNS Efficiency with Practical PIM Hardware

ReHy: A ReRAM-based Digital/Analog Hybrid PIM Architecture for Accelerating CNN Training

PyNeRF: Pyramidal Neural Radiance Fields

ACFNeRF: Accelerating and Cache-Free Neural Rendering Via Point Cloud-Based Distance Fields.

Tri-MipRF: Tri-Mip Representation for Efficient Anti-Aliasing Neural Radiance Fields