RetinaGS: Scalable Training for Dense Scene Rendering with Billion-Scale 3D Gaussians

Bingling Li,Shengyi Chen,Luchao Wang,Kaimin Liao,Sijie Yan,Yuanjun Xiong

2024-06-23

Abstract:In this work, we explore the possibility of training high-parameter 3D Gaussian splatting (3DGS) models on large-scale, high-resolution datasets. We design a general model parallel training method for 3DGS, named RetinaGS, which uses a proper rendering equation and can be applied to any scene and arbitrary distribution of Gaussian primitives. It enables us to explore the scaling behavior of 3DGS in terms of primitive numbers and training resolutions that were difficult to explore before and surpass previous state-of-the-art reconstruction quality. We observe a clear positive trend of increasing visual quality when increasing primitive numbers with our method. We also demonstrate the first attempt at training a 3DGS model with more than one billion primitives on the full MatrixCity dataset that attains a promising visual quality.

Computer Vision and Pattern Recognition,Graphics

What problem does this paper attempt to address?

The main goal of this paper is to address the efficient training of high-parameter 3D Gaussian Splatting (3DGS) models on large-scale, high-resolution datasets. Specifically, the researchers designed a general model parallel training method called RetinaGS to overcome the computational and memory limitations encountered when processing large-scale scene reconstructions.

The main contributions of the paper can be summarized as follows:

1. **Distributed Training Method**: The research team proposed a distributed training framework that can effectively allocate computational tasks to multiple GPUs, making it possible to train 3DGS models on large-scale, high-resolution datasets that are difficult to handle on a single GPU.

2. **RetinaGS Method**: By dividing the 3D space into a series of convex subspaces and assigning a portion of the Gaussian splatting (i.e., "splat") to each subspace, RetinaGS can achieve efficient distributed training while maintaining equivalence to single-GPU training. This method allows researchers to explore the training effects at larger scales of Gaussian splatting and higher resolutions.

3. **Scalability Verification**: The researchers demonstrated a significant improvement in reconstruction quality when increasing the number of Gaussian splats. Additionally, they showcased the first successful case of training on a city-level dataset (MatrixCity dataset) containing a large number of images using over one billion Gaussian splats.

4. **Experimental Results**: Validation on multiple high-resolution 3D reconstruction datasets showed that the 3DGS models trained with RetinaGS outperformed existing 3DGS methods on datasets such as Mip-NeRF360, Mega-NeRF, ScanNet++, and MatrixCity.

5. **Large-Scale Model Training**: The paper mentioned the use of 64 A100 GPUs for a 10-day training period, successfully training a 3DGS model with over one billion Gaussian splats on a dataset containing 141,652 images (totaling over 14.8 billion pixels). This scale had not been reported before.

In summary, the method proposed in this paper addresses the challenges of training 3DGS models on large-scale, high-resolution datasets and achieves unprecedented visual quality and reconstruction accuracy.