SpotlessSplats: Ignoring Distractors in 3D Gaussian Splatting

Sara Sabour,Lily Goli,George Kopanas,Mark Matthews,Dmitry Lagun,Leonidas Guibas,Alec Jacobson,David J. Fleet,Andrea Tagliasacchi

2024-07-30

Abstract:3D Gaussian Splatting (3DGS) is a promising technique for 3D reconstruction, offering efficient training and rendering speeds, making it suitable for real-time applications.However, current methods require highly controlled environments (no moving people or wind-blown elements, and consistent lighting) to meet the inter-view consistency assumption of 3DGS. This makes reconstruction of real-world captures problematic. We present SpotLessSplats, an approach that leverages pre-trained and general-purpose features coupled with robust optimization to effectively ignore transient distractors. Our method achieves state-of-the-art reconstruction quality both visually and quantitatively, on casual captures. Additional results available at: <a class="link-external link-https" href="https://spotlesssplats.github.io" rel="external noopener nofollow">this https URL</a>

Computer Vision and Pattern Recognition,Machine Learning

What problem does this paper attempt to address?

The paper primarily addresses the challenges encountered in 3D scene reconstruction using 3D Gaussian Splatting (3DGS) technology, especially when dealing with image data in non-ideal environments. Specifically, the paper investigates the following key issues:

1. **Handling Interference in Dynamic Environments**: Existing 3DGS methods require highly controlled environments, such as no moving people or wind-blown objects, and consistent lighting conditions. These limitations make it difficult to effectively reconstruct data captured in the real world.

2. **Improving Robustness**: To enable 3DGS to work better on real-world images, it is necessary to enhance its robustness against various interference factors (e.g., moving objects, lighting changes).

3. **Reducing Computational Resource Consumption**: In addition to improving reconstruction quality, the paper also focuses on how to reduce the required computational resources while ensuring reconstruction accuracy, including reducing the number of Gaussian elements used to save computation and memory.

To address the above issues, the paper proposes a method called SpotLessSplats, which achieves the following:

- **Utilizing Pre-trained Features**: Using rich features extracted from text-to-image models to detect outliers in images, rather than directly relying on color residuals.

- **Adopting an Adaptive Optimization Strategy**: Developing an adaptive, robust loss function that can reliably identify and ignore transient interferences during the reconstruction process.

- **Proposing a Sparsification Strategy**: Introducing a new sparsification method that can significantly reduce the number of Gaussian elements required without affecting reconstruction quality, thereby saving computational and memory resources.

In summary, the main contribution of this paper is to provide an efficient and robust 3D scene reconstruction framework, particularly suitable for handling real-world image data containing various interference factors.