Compact 3D Gaussian Representation for Radiance Field

Joo Chan Lee,Daniel Rho,Xiangyu Sun,Jong Hwan Ko,Eunbyung Park

2024-02-15

Abstract:Neural Radiance Fields (NeRFs) have demonstrated remarkable potential in capturing complex 3D scenes with high fidelity. However, one persistent challenge that hinders the widespread adoption of NeRFs is the computational bottleneck due to the volumetric rendering. On the other hand, 3D Gaussian splatting (3DGS) has recently emerged as an alternative representation that leverages a 3D Gaussisan-based representation and adopts the rasterization pipeline to render the images rather than volumetric rendering, achieving very fast rendering speed and promising image quality. However, a significant drawback arises as 3DGS entails a substantial number of 3D Gaussians to maintain the high fidelity of the rendered images, which requires a large amount of memory and storage. To address this critical issue, we place a specific emphasis on two key objectives: reducing the number of Gaussian points without sacrificing performance and compressing the Gaussian attributes, such as view-dependent color and covariance. To this end, we propose a learnable mask strategy that significantly reduces the number of Gaussians while preserving high performance. In addition, we propose a compact but effective representation of view-dependent color by employing a grid-based neural field rather than relying on spherical harmonics. Finally, we learn codebooks to compactly represent the geometric attributes of Gaussian by vector quantization. With model compression techniques such as quantization and entropy coding, we consistently show over 25$\times$ reduced storage and enhanced rendering speed, while maintaining the quality of the scene representation, compared to 3DGS. Our work provides a comprehensive framework for 3D scene representation, achieving high performance, fast training, compactness, and real-time rendering. Our project page is available at <a class="link-external link-https" href="https://maincold2.github.io/c3dgs/" rel="external noopener nofollow">this https URL</a>.

Computer Vision and Pattern Recognition,Graphics

What problem does this paper attempt to address?

This paper proposes a solution to the problem of excessive memory and storage consumption in the 3D Gaussian Splatting (3DGS) method. 3DGS is a fast neural radiance field rendering technique that uses 3D Gaussian distributions to represent scenes, but it requires a large number of 3D Gaussians to maintain image quality, leading to increased memory and storage requirements. The main goal of the paper is to reduce the number of Gaussian points without sacrificing performance and compress Gaussian attributes, such as disparity-related color and covariance.

To achieve these goals, the paper proposes the following methods:
1. Learning-based masking strategy: By identifying and removing redundant Gaussian points that have the least impact on overall performance, the number of Gaussian points is reduced.
2. Compressed disparity-related color: Adopting a grid-based neural field to efficiently represent color instead of relying on spherical harmonics reduces spatial complexity.
3. Codebook representation of geometric attributes: Learning to represent Gaussian geometric properties, such as scale and rotation, through vector quantization, allows for storing less data without significant loss of information.

These improvements allow the model to achieve faster training speeds, real-time rendering, and significantly reduced storage requirements while maintaining high-quality scene reconstruction. Experimental results show that the proposed compact 3D Gaussian representation reduces storage requirements by approximately 15 times compared to 3DGS, improves rendering speed, and maintains similar or better image quality. Further post-processing, such as quantization and entropy coding, can achieve compression ratios of over 25 times.