HPC: Hierarchical Progressive Coding Framework for Volumetric Video

Zihan Zheng,Houqiang Zhong,Qiang Hu,Xiaoyun Zhang,Li Song,Ya Zhang,Yanfeng Wang

DOI: https://doi.org/10.1145/3664647.3681107

2024-08-03

Abstract:Volumetric video based on Neural Radiance Field (NeRF) holds vast potential for various 3D applications, but its substantial data volume poses significant challenges for compression and transmission. Current NeRF compression lacks the flexibility to adjust video quality and bitrate within a single model for various network and device capacities. To address these issues, we propose HPC, a novel hierarchical progressive volumetric video coding framework achieving variable bitrate using a single model. Specifically, HPC introduces a hierarchical representation with a multi-resolution residual radiance field to reduce temporal redundancy in long-duration sequences while simultaneously generating various levels of detail. Then, we propose an end-to-end progressive learning approach with a multi-rate-distortion loss function to jointly optimize both hierarchical representation and compression. Our HPC trained only once can realize multiple compression levels, while the current methods need to train multiple fixed-bitrate models for different rate-distortion (RD) tradeoffs. Extensive experiments demonstrate that HPC achieves flexible quality levels with variable bitrate by a single model and exhibits competitive RD performance, even outperforming fixed-bitrate models across various datasets.

Computer Vision and Pattern Recognition,Machine Learning,Multimedia,Image and Video Processing

What problem does this paper attempt to address?

### What problems does this paper attempt to solve?

This paper aims to solve the challenges faced by volumetric video based on Neural Radiance Field (NeRF) during compression and transmission. Specifically, the current NeRF compression methods have the following problems:

1. **Lack of flexibility**: Existing compression methods cannot flexibly adjust video quality and bit rate in the same model to adapt to different network and device capacities.
2. **Low efficiency**: In order to achieve different bit - rate and distortion - rate (RD) trade - offs, existing methods need to train and store multiple models separately for each fixed bit rate, resulting in high storage costs and low efficiency.
3. **Loss of dynamic details**: Traditional image / video coding techniques fail to optimize representation and compression simultaneously, resulting in loss of dynamic details and reduced compression efficiency.

To solve these problems, the authors propose HPC (Hierarchical Progressive Coding Framework), a novel hierarchical progressive volumetric video coding framework that can achieve variable bit rates in a single model and significantly improve compression efficiency and RD performance. The main contributions of HPC include:

- **Flexible quality and bit - rate adjustment**: By introducing the multi - resolution residual radiance field, HPC can achieve progressive encoding and decoding at different quality levels in a single model.
- **Efficient hierarchical representation**: HPC uses the multi - resolution residual radiance field with low temporal redundancy to efficiently represent volumetric video, thereby reducing the amount of data and improving compression efficiency.
- **End - to - end progressive learning**: By jointly optimizing the hierarchical representation and compression, HPC introduces a multi - rate - distortion loss function and a step - by - step training strategy to enhance the RD performance of each layer and the overall.

In summary, HPC solves the key problems in existing NeRF compression methods by improving coding efficiency and flexibility, providing a new solution for efficient transmission and high - quality reconstruction of volumetric video.