Abstract:3D Gaussian Splatting (3DGS) has emerged as a mainstream for novel view synthesis, leveraging continuous aggregations of Gaussian functions to model scene geometry. However, 3DGS suffers from substantial memory requirements to store the multitude of Gaussians, hindering its practicality. To address this challenge, we introduce GaussianSpa, an optimization-based simplification framework for compact and high-quality 3DGS. Specifically, we formulate the simplification as an optimization problem associated with the 3DGS training. Correspondingly, we propose an efficient "optimizing-sparsifying" solution that alternately solves two independent sub-problems, gradually imposing strong sparsity onto the Gaussians in the training process. Our comprehensive evaluations on various datasets show the superiority of GaussianSpa over existing state-of-the-art approaches. Notably, GaussianSpa achieves an average PSNR improvement of 0.9 dB on the real-world Deep Blending dataset with 10$\times$ fewer Gaussians compared to the vanilla 3DGS. Our project page is available at <a class="link-external link-https" href="https://gaussianspa.github.io/" rel="external noopener nofollow">this https URL</a>.
What problem does this paper attempt to address?
### Problems the paper attempts to solve
The paper aims to solve the problem of excessive memory consumption faced by 3D Gaussian Splatting (3DGS) in novel view synthesis. Although 3DGS is significantly superior to traditional NeRF methods in terms of rendering quality and computational efficiency, it needs to store a large number of Gaussian points to represent complex scenes, resulting in a huge amount of memory usage, exceeding the capacity limits of typical hardware. This makes it difficult to process high - resolution scenes and limits its application in resource - constrained environments.
To solve this problem, the paper proposes an optimization - based simplification framework - GaussianSpa. This framework gradually reduces the number of Gaussian points through an optimization process while maintaining high - quality rendering effects. Specifically, GaussianSpa models the 3DGS simplification problem as an optimization problem and alternately solves two independent sub - problems during the training process, gradually imposing strong sparsity constraints on the Gaussian points. In this way, GaussianSpa can significantly reduce the number of Gaussian points while maximizing the retention of the information of the original Gaussian points, thereby achieving a compact and high - quality 3DGS model.
### Main contributions
1. **Universal 3DGS simplification framework**:
- Model the 3DGS simplification objective as an optimization problem and gradually impose sparsity constraints during the 3DGS training process without explicitly removing a specific number of Gaussian points. This method can maximize the preservation and smooth transfer of the information of the original model to the sparse Gaussian points.
2. **Efficient "optimization - sparsification" solution**:
- Propose an efficient "optimization - sparsification" solution that can be integrated into the 3DGS training process with almost no additional cost. This solution solves two sub - problems respectively: optimize the original loss function and attach a regularization term in the "optimization" step; project the auxiliary Gaussian points to the constrained sparse space in the "sparsification" step.
3. **Extensive experimental verification**:
- Verify the superiority of GaussianSpa in rendering quality through extensive experiments on multiple complex scenes. In particular, compared with traditional 3DGS, GaussianSpa has an average PSNR improvement of 0.4 dB when the number of Gaussian points is reduced by 10 times, and an improvement of 0.9 dB on the Deep Blending dataset. In addition, GaussianSpa performs well in high - quality rendering of details and sparse 3D Gaussian views.
### Method overview
1. **Background**:
- 3D Gaussian Splatting (3DGS) explicitly represents a scene through a set of continuous 3D Gaussian points, and each Gaussian point is described by parameters such as covariance matrix, center position, transparency, and spherical harmonic coefficients. During the rendering process, 3DGS projects 3D Gaussian points onto a 2D plane and calculates the color value of each pixel.
2. **Problem modeling**:
- In order to reduce irreversible information loss, GaussianSpa models the 3DGS simplification problem as a constrained optimization problem by introducing sparsity constraints. Specifically, the optimization objective is to make the number of Gaussian points not exceed the target number while maintaining the rendering quality.
3. **Optimization - sparsification solution**:
- By introducing auxiliary variables and Lagrange multipliers, transform the constrained optimization problem into an unconstrained optimization problem. Then, gradually reduce the number of Gaussian points by alternately performing the "optimization" step and the "sparsification" step while maintaining high - quality rendering effects.
4. **Overall process**:
- During the 3DGS training process, gradually reduce the number of Gaussian points through the "optimization - sparsification" steps. Finally, remove all "zero" Gaussian points and perform a slight fine - tuning to further improve performance and obtain a compact and high - quality 3DGS model.
### Experimental results
- **Quantitative results**:
- Experimental results on multiple datasets show that GaussianSpa can still maintain or even improve the rendering quality when the number of Gaussian points is significantly reduced. For example, on the Mip - NeRF 360, Tanks & Temples, and Deep Blending datasets, GaussianSpa achieves PSNR improvements of 0.4 dB, 0.4 dB, and 0.9 dB respectively.
- **Qualitative results**:
- Through visual analysis,