3D Gaussian Splatting as Markov Chain Monte Carlo

Shakiba Kheradmand,Daniel Rebain,Gopal Sharma,Weiwei Sun,Jeff Tseng,Hossam Isack,Abhishek Kar,Andrea Tagliasacchi,Kwang Moo Yi

2024-06-17

Abstract:While 3D Gaussian Splatting has recently become popular for neural rendering, current methods rely on carefully engineered cloning and splitting strategies for placing Gaussians, which can lead to poor-quality renderings, and reliance on a good initialization. In this work, we rethink the set of 3D Gaussians as a random sample drawn from an underlying probability distribution describing the physical representation of the scene-in other words, Markov Chain Monte Carlo (MCMC) samples. Under this view, we show that the 3D Gaussian updates can be converted as Stochastic Gradient Langevin Dynamics (SGLD) updates by simply introducing noise. We then rewrite the densification and pruning strategies in 3D Gaussian Splatting as simply a deterministic state transition of MCMC samples, removing these heuristics from the framework. To do so, we revise the 'cloning' of Gaussians into a relocalization scheme that approximately preserves sample probability. To encourage efficient use of Gaussians, we introduce a regularizer that promotes the removal of unused Gaussians. On various standard evaluation scenes, we show that our method provides improved rendering quality, easy control over the number of Gaussians, and robustness to initialization.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The paper primarily addresses and improves upon the issues present in the 3D Gaussian Splatting (3DGS) method. 3DGS is a technique used for neural rendering, which approximates a scene by placing a series of 3D Gaussian distributions and utilizes these Gaussians for efficient rendering. However, existing 3DGS methods rely on complex cloning and splitting strategies when placing Gaussian distributions, which can lead to poor rendering quality and sensitivity to initialization.

The main contributions of the paper are:

1. **Linking 3DGS with Markov Chain Monte Carlo (MCMC) sampling**: The authors re-examined the update rules for Gaussian distributions in 3DGS and viewed them as part of a Stochastic Gradient Langevin Dynamics (SGLD) update rule, where noise is introduced to facilitate sample exploration. This perspective aligns the updates in 3DGS more closely with the MCMC framework, thereby naturally exploring the scene representation.

2. **Replacing the original cloning and splitting strategy**: The original strategy required careful tuning and was sensitive to initialization. The paper proposes a new strategy based on relocation, by moving "dead" Gaussian distributions to the positions of "live" Gaussian distributions, while ensuring minimal impact on rendering. This avoids the issues present in the original strategy.

3. **Introducing regularization to improve efficiency**: To encourage the efficient use of Gaussian distributions, the paper introduces an L1 regularization term to promote the disappearance of unnecessary Gaussian distributions.

4. **Enhancing robustness to initialization**: The new method significantly improves robustness to initialization, achieving high-quality rendering results even when starting from random initialization.

5. **Improving rendering quality**: Experimental results show that the new method not only provides higher rendering quality than the original 3DGS but also better controls the number of Gaussian distributions while reducing dependency on initialization.

In summary, this paper places 3DGS within the MCMC framework, proposes improved update rules and strategies, addresses the issues in existing 3DGS methods, and significantly enhances rendering quality and robustness to initialization.