Abstract:The neural radiance fields (NeRF) have advanced the development of 3D volumetric video technology, but the large data volumes they involve pose significant challenges for storage and transmission. To address these problems, the existing solutions typically compress these NeRF representations after the training stage, leading to a separation between representation training and compression. In this paper, we try to directly learn a compact NeRF representation for volumetric video in the training stage based on the proposed rate-aware compression framework. Specifically, for volumetric video, we use a simple yet effective modeling strategy to reduce temporal redundancy for the NeRF representation. Then, during the training phase, an implicit entropy model is utilized to estimate the bitrate of the NeRF representation. This entropy model is then encoded into the bitstream to assist in the decoding of the NeRF representation. This approach enables precise bitrate estimation, thereby leading to a compact NeRF representation. Furthermore, we propose an adaptive quantization strategy and learn the optimal quantization step for the NeRF representations. Finally, the NeRF representation can be optimized by using the rate-distortion trade-off. Our proposed compression framework can be used for different representations and experimental results demonstrate that our approach significantly reduces the storage size with marginal distortion and achieves state-of-the-art rate-distortion performance for volumetric video on the HumanRF and ReRF datasets. Compared to the previous state-of-the-art method TeTriRF, we achieved an approximately -80% BD-rate on the HumanRF dataset and -60% BD-rate on the ReRF dataset.

Neural Radiance Field-Assisted Static-Scene Video Coding

Static-Scene Video Compression with Neural Radiance Fields

NeRFCodec: Neural Feature Compression Meets Neural Radiance Fields for Memory-Efficient Scene Representation

Neural NeRF Compression

Efficient Dynamic-NeRF Based Volumetric Video Coding with Rate Distortion Optimization

VRVVC: Variable-Rate NeRF-Based Volumetric Video Compression

JointRF: End-to-End Joint Optimization for Dynamic Neural Radiance Field Representation and Compression

Rate-aware Compression for NeRF-based Volumetric Video

CodecNeRF: Toward Fast Encoding and Decoding, Compact, and High-quality Novel-view Synthesis

HPC: Hierarchical Progressive Coding Framework for Volumetric Video

Neural Video Compression with Re-Parametrisation Scene Content-Adaptive Network

3DST: A Robust End-to-End 3D Scene Transmission Framework Using Neural Radiance Fields

NeVRF: Neural Video-based Radiance Fields for Long-duration Sequences

Neural 3D Scene Compression via Model Compression

ECRF: Entropy-Constrained Neural Radiance Fields Compression with Frequency Domain Optimization

Resolution-Agnostic Neural Compression for High-Fidelity Portrait Video Conferencing via Implicit Radiance Fields

SPC-NeRF: Spatial Predictive Compression for Voxel Based Radiance Field

Neural Residual Radiance Fields for Streamably Free-Viewpoint Videos

Hyb-NeRF: A Multiresolution Hybrid Encoding for Neural Radiance Fields

Semantic Neural Rendering-based Video Coding: Towards Ultra-Low Bitrate Video Conferencing

Ced-NeRF: A Compact and Efficient Method for Dynamic Neural Radiance Fields