NGP-RT: Fusing Multi-Level Hash Features with Lightweight Attention for Real-Time Novel View Synthesis

Yubin Hu,Xiaoyang Guo,Yang Xiao,Jingwei Huang,Yong-Jin Liu

2024-07-15

Abstract:This paper presents NGP-RT, a novel approach for enhancing the rendering speed of Instant-NGP to achieve real-time novel view synthesis. As a classic NeRF-based method, Instant-NGP stores implicit features in multi-level grids or hash tables and applies a shallow MLP to convert the implicit features into explicit colors and densities. Although it achieves fast training speed, there is still a lot of room for improvement in its rendering speed due to the per-point MLP executions for implicit multi-level feature aggregation, especially for real-time applications. To address this challenge, our proposed NGP-RT explicitly stores colors and densities as hash features, and leverages a lightweight attention mechanism to disambiguate the hash collisions instead of using computationally intensive MLP. At the rendering stage, NGP-RT incorporates a pre-computed occupancy distance grid into the ray marching strategy to inform the distance to the nearest occupied voxel, thereby reducing the number of marching points and global memory access. Experimental results show that on the challenging Mip-NeRF360 dataset, NGP-RT achieves better rendering quality than previous NeRF-based methods, achieving 108 fps at 1080p resolution on a single Nvidia RTX 3090 GPU. Our approach is promising for NeRF-based real-time applications that require efficient and high-quality rendering.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The problem that this paper attempts to solve is to achieve real - time novel view synthesis based on Neural Radiance Field (NeRF) while maintaining high - quality rendering. Specifically, although existing NeRF methods perform well in terms of training speed and rendering quality, they still have difficulty meeting the requirements of real - time rendering in large - scale scenes. In particular, the Instant - NGP method achieves high - quality novel view synthesis through multi - level implicit feature grids and shallow MLPs, but the shallow MLPs for per - point feature aggregation still result in a relatively slow rendering speed.

To solve this problem, the paper proposes the NGP - RT method, which mainly improves the rendering speed in the following two aspects:

1. **Lightweight Attention Mechanism**: NGP - RT explicitly stores colors and densities as hash features and utilizes a lightweight attention mechanism to eliminate hash collisions instead of using computationally intensive MLPs. This method can efficiently aggregate multi - level explicit hash features while maintaining high expressiveness.

2. **Occupancy - Distance Grid**: During the rendering stage, NGP - RT introduces a pre - computed occupancy - distance grid and incorporates it into the ray - marching strategy. The occupancy - distance grid can inform the distance to the nearest occupied voxel, thereby reducing the number of marching points and the number of global memory accesses, further accelerating the rendering process.

Through these improvements, NGP - RT achieves better rendering quality and a higher frame rate (reaching 108 fps at 1080p resolution) than existing methods on the challenging Mip - NeRF 360 dataset. This makes NGP - RT have great potential in real - time applications that require efficient and high - quality rendering.