End-to-End Rate-Distortion Optimized 3D Gaussian Representation

Henan Wang,Hanxin Zhu,Tianyu He,Runsen Feng,Jiajun Deng,Jiang Bian,Zhibo Chen

2024-10-21

Abstract:3D Gaussian Splatting (3DGS) has become an emerging technique with remarkable potential in 3D representation and image rendering. However, the substantial storage overhead of 3DGS significantly impedes its practical applications. In this work, we formulate the compact 3D Gaussian learning as an end-to-end Rate-Distortion Optimization (RDO) problem and propose RDO-Gaussian that can achieve flexible and continuous rate control. RDO-Gaussian addresses two main issues that exist in current schemes: 1) Different from prior endeavors that minimize the rate under the fixed distortion, we introduce dynamic pruning and entropy-constrained vector quantization (ECVQ) that optimize the rate and distortion at the same time. 2) Previous works treat the colors of each Gaussian equally, while we model the colors of different regions and materials with learnable numbers of parameters. We verify our method on both real and synthetic scenes, showcasing that RDO-Gaussian greatly reduces the size of 3D Gaussian over 40x, and surpasses existing methods in rate-distortion performance.

Computer Vision and Pattern Recognition,Graphics

What problem does this paper attempt to address?

The problem that this paper attempts to solve is the huge storage overhead problem faced by 3D Gaussian Splatting (3DGS) in practical applications. Although 3DGS has shown significant potential in 3D representation and image rendering, its large storage requirements seriously hinder its wide application.

To solve this problem, the author proposes an end - to - end Rate - Distortion Optimization (RDO) framework, named RDO - Gaussian. Specifically, this method aims to achieve flexible and continuous rate control through the following two main improvements:

1. **Joint optimization of rate and distortion**: Different from previous methods that minimize the rate under a fixed distortion, RDO - Gaussian introduces dynamic pruning and Entropy - Constrained Vector Quantization (ECVQ) to optimize the rate and distortion simultaneously.

2. **Adaptive modeling of the colors of different regions and materials**: Previous works treat the color of each Gaussian in the same way, while RDO - Gaussian learns different numbers of parameters according to different regions and materials, thus more efficiently handling color representation under different materials and lighting conditions.

Through these improvements, RDO - Gaussian can not only significantly reduce the size of the 3D Gaussian representation (more than 40 times), but also outperform existing methods in rate - distortion performance. In addition, the experimental results in real and synthetic scenes verify its effectiveness.

### Formula summary

- Rate - distortion optimization objective function:
\[
L_{\text{total}}=\lambda_{\text{GSprune}}L_{\text{GSprune}}+\lambda_{\text{SHprune}}L_{\text{SHprune}}+L_{\text{rate}}+L_{\text{VQ}}+L_{\text{render}}
\]

- Gaussian pruning loss:
\[
L_{\text{GSprune}}=\frac{1}{N}\sum_{i}\phi^{\text{soft}}_i
\]

- Adaptive spherical harmonic function pruning loss:
\[
L_{\text{SHprune}}=\frac{1}{N}\sum_{i}\sum_{l = 1}^{k}\frac{2l + 1}{(k + 1)^2-1}\theta^{(l),\text{soft}}_i
\]

- Rate loss and quantization loss of Entropy - Constrained Vector Quantization (ECVQ):
\[
L_{\text{rate}}=\frac{1}{N}\sum_{i = 1}^{N}\left(\frac{r^{(s)}_{i,j}}{\lambda^{(s)}}+\frac{r^{(r)}_{i,j}}{\lambda^{(r)}}+\frac{r^{(\text{DC})}_{i,j}}{\lambda^{(\text{DC})}}+\frac{r^{(\text{SH1})}_{i,j}}{\lambda^{(\text{SH1})}}+\frac{r^{(\text{SH2})}_{i,j}}{\lambda^{(\text{SH2})}}+\frac{r^{(\text{SH3})}_{i,j}}{\lambda^{(\text{SH3})}}\right)
\]
\[
L_{\text{VQ}}=\frac{1}{N}\sum_{i = 1}^{N}\left(d^{(s)}_{i,j}+d^{(r)}_{i,j}+d^{(\text{DC})}_{i,j}+d^{(\text{SH1})}_{i}
\]