Abstract:Despite the photorealistic novel view synthesis (NVS) performance achieved by the original 3D Gaussian splatting (3DGS), its rendering quality significantly degrades with sparse input views. This performance drop is mainly caused by the limited number of initial points generated from the sparse input, insufficient supervision during the training process, and inadequate regularization of the oversized Gaussian ellipsoids. To handle these issues, we propose the LoopSparseGS, a loop-based 3DGS framework for the sparse novel view synthesis task. In specific, we propose a loop-based Progressive Gaussian Initialization (PGI) strategy that could iteratively densify the initialized point cloud using the rendered pseudo images during the training process. Then, the sparse and reliable depth from the Structure from Motion, and the window-based dense monocular depth are leveraged to provide precise geometric supervision via the proposed Depth-alignment Regularization (DAR). Additionally, we introduce a novel Sparse-friendly Sampling (SFS) strategy to handle oversized Gaussian ellipsoids leading to large pixel errors. Comprehensive experiments on four datasets demonstrate that LoopSparseGS outperforms existing state-of-the-art methods for sparse-input novel view synthesis, across indoor, outdoor, and object-level scenes with various image resolutions.

What problem does this paper attempt to address?

The paper mainly proposes solutions to the problems encountered by 3D Gaussian Splatting (3DGS) in novel view synthesis (NVS) under sparse view scenarios. ### Research Background and Problems - **3D Gaussian Splatting**: This is an effective technique for NVS that can efficiently simulate the highly detailed appearance and geometry of 3D scenes. However, its rendering quality significantly degrades when the number of input images is limited. - **Challenges Brought by Sparse Input**: - Insufficient Initial Gaussian Points: The initial Gaussian points provided by Structure from Motion (SfM) may be too sparse. - Limited Geometric Reconstruction: Sparse input leads to an under-constrained and ill-posed problem for scene appearance and geometric reconstruction. - Overly Large Gaussian Ellipsoids: During optimization, the scale of some Gaussian ellipsoids becomes too large, leading to overfitting issues. ### Solution Overview The paper proposes the LoopSparseGS framework, a 3DGS framework based on a loop mechanism, aimed at addressing the above issues: 1. **Progressive Gaussian Initialization (PGI)**: Uses an iterative approach to increase the density of the initial 3D point cloud by generating additional initial points using the structure from motion method with a combination of training images and high-quality pseudo-images. 2. **Depth-alignment Regularization (DAR)**: Combines sparse but accurate depth information obtained from SfM with dense but relatively scaled depth information obtained from a monocular depth estimation model. Depth alignment is performed using a sliding window approach to provide effective geometric constraints. 3. **Sparse-friendly Sampling (SFS)**: Proposes a pixel error-based strategy to identify and segment overly large Gaussian ellipsoids, improving the representation capability of large pixel areas. ### Main Contributions - Proposes a new 3DGS framework, LoopSparseGS, which is particularly suitable for novel view synthesis tasks with sparse input, featuring Gaussian initialization and precise geometric constraints under a loop mechanism. - Develops the Progressive Gaussian Initialization method, which increases the initial Gaussian points by iteratively fusing rendered images and training images. - Proposes the Depth Alignment Regularization method, which provides smoother and more accurate rendering depth by aligning relatively scaled monocular depth with absolute scaled SfM-derived depth. - Introduces the Sparse-friendly Sampling strategy to address the issue of overly large Gaussian ellipsoids in sparse input scenarios, thereby improving the quality of view synthesis. - Experimental results show that the proposed LoopSparseGS outperforms existing state-of-the-art methods on four different datasets, including indoor scenes, outdoor scenes, and object-level scenes of different resolutions.

LoopSparseGS: Loop Based Sparse-View Friendly Gaussian Splatting

SparseGS: Real-Time 360° Sparse View Synthesis using Gaussian Splatting

CoherentGS: Sparse Novel View Synthesis with Coherent 3D Gaussians

SparseLGS: Sparse View Language Embedded Gaussian Splatting

FewViewGS: Gaussian Splatting with Few View Matching and Multi-stage Training

Structure Consistent Gaussian Splatting with Matching Prior for Few-shot Novel View Synthesis

Binocular-Guided 3D Gaussian Splatting with View Consistency for Sparse View Synthesis

HybridGS: Decoupling Transients and Statics with 2D and 3D Gaussian Splatting

MVGS: Multi-view-regulated Gaussian Splatting for Novel View Synthesis

A Construct-Optimize Approach to Sparse View Synthesis without Camera Pose

LM-Gaussian: Boost Sparse-view 3D Gaussian Splatting with Large Model Priors

FSGS: Real-Time Few-shot View Synthesis using Gaussian Splatting

Optimizing 3D Gaussian Splatting for Sparse Viewpoint Scene Reconstruction

InstantSplat: Sparse-view SfM-free Gaussian Splatting in Seconds

MCGS: Multiview Consistency Enhancement for Sparse-View 3D Gaussian Radiance Fields

MVPGS: Excavating Multi-view Priors for Gaussian Splatting from Sparse Input Views

Pseudo-Viewpoint Regularized 3D Gaussian Splatting for Remote Sensing Few-Shot Novel View Synthesis

SLGaussian: Fast Language Gaussian Splatting in Sparse Views

3DGS-Enhancer: Enhancing Unbounded 3D Gaussian Splatting with View-consistent 2D Diffusion Priors

TSGaussian: Semantic and Depth-Guided Target-Specific Gaussian Splatting from Sparse Views

RSGaussian:3D Gaussian Splatting with LiDAR for Aerial Remote Sensing Novel View Synthesis