Neural NeRF Compression

Tuan Pham,Stephan Mandt

2024-06-13

Abstract:Neural Radiance Fields (NeRFs) have emerged as powerful tools for capturing detailed 3D scenes through continuous volumetric representations. Recent NeRFs utilize feature grids to improve rendering quality and speed; however, these representations introduce significant storage overhead. This paper presents a novel method for efficiently compressing a grid-based NeRF model, addressing the storage overhead concern. Our approach is based on the non-linear transform coding paradigm, employing neural compression for compressing the model's feature grids. Due to the lack of training data involving many i.i.d scenes, we design an encoder-free, end-to-end optimized approach for individual scenes, using lightweight decoders. To leverage the spatial inhomogeneity of the latent feature grids, we introduce an importance-weighted rate-distortion objective and a sparse entropy model employing a masking mechanism. Our experimental results validate that our proposed method surpasses existing works in terms of grid-based NeRF compression efficacy and reconstruction quality.

Computer Vision and Pattern Recognition,Machine Learning

What problem does this paper attempt to address?

The paper primarily addresses the storage overhead issue brought by feature grid representation in Neural Radiance Fields (NeRF) models.

Specifically, NeRF, as a powerful tool for 3D scene capture, reconstructs detailed 3D scenes through continuous volumetric representation and can synthesize new viewpoints from limited 2D data. Some recent NeRF methods utilize feature grids to improve rendering quality and speed, but this introduces significant storage overhead. Therefore, this paper proposes an efficient compression method for grid-based NeRF models to address the storage overhead issue.

The proposed method is based on a nonlinear transform coding paradigm, using neural network compression techniques to compress the feature grids in the model. This method does not require an encoder but directly optimizes the latent variables and a lightweight decoder, while also considering the spatial non-uniformity of the feature grids. It introduces an importance-weighted rate-distortion objective and a sparse entropy model. Experimental results validate that this method surpasses existing works in terms of grid-based NeRF compression efficiency and reconstruction quality.

In short, the main goal of the paper is to reduce the storage cost of grid-based NeRF models while maintaining their reconstruction quality.