3DGS-LM: Faster Gaussian-Splatting Optimization with Levenberg-Marquardt

Lukas Höllein,Aljaž Božič,Michael Zollhöfer,Matthias Nießner

2024-09-20

Abstract:We present 3DGS-LM, a new method that accelerates the reconstruction of 3D Gaussian Splatting (3DGS) by replacing its ADAM optimizer with a tailored Levenberg-Marquardt (LM). Existing methods reduce the optimization time by decreasing the number of Gaussians or by improving the implementation of the differentiable rasterizer. However, they still rely on the ADAM optimizer to fit Gaussian parameters of a scene in thousands of iterations, which can take up to an hour. To this end, we change the optimizer to LM that runs in conjunction with the 3DGS differentiable rasterizer. For efficient GPU parallization, we propose a caching data structure for intermediate gradients that allows us to efficiently calculate Jacobian-vector products in custom CUDA kernels. In every LM iteration, we calculate update directions from multiple image subsets using these kernels and combine them in a weighted mean. Overall, our method is 30% faster than the original 3DGS while obtaining the same reconstruction quality. Our optimization is also agnostic to other methods that acclerate 3DGS, thus enabling even faster speedups compared to vanilla 3DGS.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The paper aims to address the issue of prolonged optimization time in the 3D Gaussian Splatting (3DGS) reconstruction process. Specifically:

- **Main Objective**: To accelerate the 3DGS reconstruction speed by replacing the existing ADAM optimizer with a customized Levenberg-Marquardt (LM) optimizer. The proposed method reduces the optimization time by an average of 30% while maintaining the same reconstruction quality.

- **Specific Problem**: Traditional 3DGS methods require thousands of iterations to converge when processing high-resolution real-world scene datasets, which can take up to an hour. Although existing methods have attempted to shorten optimization time by reducing the number of Gaussian points or improving the implementation of differentiable rasterizers, they still rely on the ADAM optimizer for iterative optimization, resulting in a lengthy process.

- **Solution**: This paper proposes a new method, 3DGS-LM, which introduces a customized Levenberg-Marquardt algorithm into the 3DGS reconstruction process, combined with an efficient GPU parallelization scheme to accelerate the optimization process. Additionally, a caching data structure is designed for efficient computation of Jacobian matrix-vector products, further enhancing optimization efficiency.

In summary, the main goal of the paper is to significantly reduce the time required for 3DGS reconstruction by improving the optimizer, while maintaining reconstruction quality comparable to existing methods.