Taming 3DGS: High-Quality Radiance Fields with Limited Resources

Saswat Subhajyoti Mallick,Rahul Goel,Bernhard Kerbl,Francisco Vicente Carrasco,Markus Steinberger,Fernando De La Torre

2024-06-22

Abstract:3D Gaussian Splatting (3DGS) has transformed novel-view synthesis with its fast, interpretable, and high-fidelity rendering. However, its resource requirements limit its usability. Especially on constrained devices, training performance degrades quickly and often cannot complete due to excessive memory consumption of the model. The method converges with an indefinite number of Gaussians -- many of them redundant -- making rendering unnecessarily slow and preventing its usage in downstream tasks that expect fixed-size inputs. To address these issues, we tackle the challenges of training and rendering 3DGS models on a budget. We use a guided, purely constructive densification process that steers densification toward Gaussians that raise the reconstruction quality. Model size continuously increases in a controlled manner towards an exact budget, using score-based densification of Gaussians with training-time priors that measure their contribution. We further address training speed obstacles: following a careful analysis of 3DGS' original pipeline, we derive faster, numerically equivalent solutions for gradient computation and attribute updates, including an alternative parallelization for efficient backpropagation. We also propose quality-preserving approximations where suitable to reduce training time even further. Taken together, these enhancements yield a robust, scalable solution with reduced training times, lower compute and memory requirements, and high quality. Our evaluation shows that in a budgeted setting, we obtain competitive quality metrics with 3DGS while achieving a 4--5x reduction in both model size and training time. With more generous budgets, our measured quality surpasses theirs. These advances open the door for novel-view synthesis in constrained environments, e.g., mobile devices.

Computer Vision and Pattern Recognition,Graphics

What problem does this paper attempt to address?

This paper mainly solves the efficiency and quality issues of the 3D Gaussian Splatting (3DGS) method under limited resources. 3DGS is a technique used for novel view synthesis, which is known for its fast, interpretable, and high-fidelity rendering. However, its practicality is constrained by its resource requirements. On resource-constrained devices, training performance quickly deteriorates, and the memory consumption of the model may prevent training from completing. The optimization process often generates a large number of redundant Gaussians, resulting in slower rendering speed and not suitable for downstream tasks with fixed input sizes.

To address these problems, the paper proposes a method for training and rendering the 3DGS model within a budget. By guiding a purely constructive densification process, densification is directed towards Gaussians that can improve reconstruction quality. The model size gradually increases in a controlled manner until it reaches the precise budget, leveraging priors during training to evaluate the contribution of Gaussians. Additionally, the paper analyzes the original pipeline of 3DGS and proposes equivalent solutions for faster gradient computation and attribute updates, as well as quality-preserving approximation methods to reduce training time when appropriate.

These improvements collectively provide a robust and scalable solution, reducing training time, lowering computational and memory requirements, while maintaining high quality. Evaluation shows that under the budget settings, this method achieves a 4-5 times reduction in model size and training time compared to 3DGS, while surpassing the measured quality of 3DGS under looser budgets. These advancements pave the way for novel view synthesis in resource-constrained environments, such as mobile devices.