LP-3DGS: Learning to Prune 3D Gaussian Splatting

Zhaoliang Zhang,Tianchen Song,Yongjae Lee,Li Yang,Cheng Peng,Rama Chellappa,Deliang Fan

2024-05-29

Abstract:Recently, 3D Gaussian Splatting (3DGS) has become one of the mainstream methodologies for novel view synthesis (NVS) due to its high quality and fast rendering speed. However, as a point-based scene representation, 3DGS potentially generates a large number of Gaussians to fit the scene, leading to high memory usage. Improvements that have been proposed require either an empirical and preset pruning ratio or importance score threshold to prune the point cloud. Such hyperparamter requires multiple rounds of training to optimize and achieve the maximum pruning ratio, while maintaining the rendering quality for each scene. In this work, we propose learning-to-prune 3DGS (LP-3DGS), where a trainable binary mask is applied to the importance score that can find optimal pruning ratio automatically. Instead of using the traditional straight-through estimator (STE) method to approximate the binary mask gradient, we redesign the masking function to leverage the Gumbel-Sigmoid method, making it differentiable and compatible with the existing training process of 3DGS. Extensive experiments have shown that LP-3DGS consistently produces a good balance that is both efficient and high quality.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The problem this paper attempts to address is the high memory consumption of 3D Gaussian Splatting (3DGS) in scene representation due to the need for a large number of Gaussian points. Traditional improvement methods rely on preset pruning ratios or importance score thresholds to prune point clouds. These methods require multiple training sessions to optimize and find the maximum pruning ratio while maintaining the rendering quality of each scene. This paper proposes a method called Learning Pruning 3DGS (LP-3DGS), which automatically finds the optimal pruning ratio by applying a trainable binary mask. This method avoids the manual adjustment of pruning thresholds required in traditional methods and achieves adaptive learning of the optimal pruning ratio for different scenes in a single training session. Additionally, the method redesigns the mask function, utilizing the Gumbel-Sigmoid method to make it differentiable and compatible with the existing 3DGS training process. Experimental results show that LP-3DGS can effectively compress the model size while ensuring rendering quality.