SplatFields: Neural Gaussian Splats for Sparse 3D and 4D Reconstruction

Marko Mihajlovic,Sergey Prokudin,Siyu Tang,Robert Maier,Federica Bogo,Tony Tung,Edmond Boyer

2024-09-17

Abstract:Digitizing 3D static scenes and 4D dynamic events from multi-view images has long been a challenge in computer vision and graphics. Recently, 3D Gaussian Splatting (3DGS) has emerged as a practical and scalable reconstruction method, gaining popularity due to its impressive reconstruction quality, real-time rendering capabilities, and compatibility with widely used visualization tools. However, the method requires a substantial number of input views to achieve high-quality scene reconstruction, introducing a significant practical bottleneck. This challenge is especially severe in capturing dynamic scenes, where deploying an extensive camera array can be prohibitively costly. In this work, we identify the lack of spatial autocorrelation of splat features as one of the factors contributing to the suboptimal performance of the 3DGS technique in sparse reconstruction settings. To address the issue, we propose an optimization strategy that effectively regularizes splat features by modeling them as the outputs of a corresponding implicit neural field. This results in a consistent enhancement of reconstruction quality across various scenarios. Our approach effectively handles static and dynamic cases, as demonstrated by extensive testing across different setups and scene complexities.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

### What problem does this paper attempt to solve?

The main goal of this paper is to improve the quality of three-dimensional (3D) and four-dimensional (4D) reconstruction in sparse multi-view scenarios. Specifically, the paper focuses on the following points:

1. **Problems with existing methods**:
- Current 3D Gaussian Splatting (3DGS) methods tend to overfit the training data when dealing with sparse input views, resulting in poor reconstruction quality.
- In dynamic scenes (4D), deploying a large number of cameras is very costly, and existing 3DGS methods perform poorly in such cases.

2. **Proposed method**:
- To overcome these issues, the authors propose the SplatFields method, which predicts Gaussian point cloud features through implicit neural fields, thereby introducing spatial autocorrelation to enhance reconstruction quality.
- SplatFields utilizes convolutional neural networks to generate triplane representations and combines them with multi-layer perceptrons (MLP) for feature regression, thereby constraining the behavior of Gaussian point clouds to be more stable in sparse view environments.

3. **Experimental validation**:
- The paper demonstrates the superior performance of SplatFields in sparse view environments through experiments in both static and dynamic scenes, particularly in novel view synthesis.
- Experimental results show that SplatFields outperforms existing 3DGS methods and other baseline methods in scenarios of varying complexity and sparsity.

In summary, this paper aims to improve the reconstruction quality of 3DGS methods under sparse view conditions by introducing spatial autocorrelation and extending it to dynamic scenes (4D).