Abstract:The field of novel view synthesis from images has seen rapid advancements with the introduction of Neural Radiance Fields (NeRF) and more recently with 3D Gaussian Splatting. Gaussian Splatting became widely adopted due to its efficiency and ability to render novel views accurately. While Gaussian Splatting performs well when a sufficient amount of training images are available, its unstructured explicit representation tends to overfit in scenarios with sparse input images, resulting in poor rendering performance. To address this, we present a 3D Gaussian-based novel view synthesis method using sparse input images that can accurately render the scene from the viewpoints not covered by the training images. We propose a multi-stage training scheme with matching-based consistency constraints imposed on the novel views without relying on pre-trained depth estimation or diffusion models. This is achieved by using the matches of the available training images to supervise the generation of the novel views sampled between the training frames with color, geometry, and semantic losses. In addition, we introduce a locality preserving regularization for 3D Gaussians which removes rendering artifacts by preserving the local color structure of the scene. Evaluation on synthetic and real-world datasets demonstrates competitive or superior performance of our method in few-shot novel view synthesis compared to existing state-of-the-art methods.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is how to accurately synthesize new - view images with only a small number of input images. Specifically, in view of the problem that the existing 3D Gaussian Splatting method is prone to over - fitting when dealing with sparse input images, resulting in poor rendering performance, the paper proposes a new - view synthesis method based on 3D Gaussian Splatting - FewViewGS. This method aims to improve the quality and accuracy of new - view synthesis under the condition of sparse image input by introducing multi - stage training schemes, new - view consistency loss under matching constraints, and local - preserving regularization techniques. ### Main contributions of the paper 1. **Proposed a few - sample new - view synthesis system**: This system is based on 3D Gaussian Splatting and can achieve high - quality new - view synthesis with only a small number of training images. 2. **Multi - stage training scheme**: Gradually optimize the scene representation through pre - training, intermediate stage, and tuning stage to ensure smoother knowledge transfer from known views to new views. 3. **New - view consistency constraint**: Utilize the correspondence between known views to supervise the generation of new views through geometric, color, and semantic losses, ensuring the consistency of the synthesized image in the overlapping area. 4. **Local - preserving regularization**: Eliminate the visual artifacts common in the few - sample case by regularizing the color parameters of 3D Gaussian points. ### Method overview - **Pre - training stage**: Optimize 3D Gaussian points only using training views to obtain the basic scene representation and depth map. - **Intermediate stage**: Focus on optimizing new views, generate unseen views using multi - view geometry and new - view interpolation sampling, and impose consistency loss by matching pixels. - **Tuning stage**: Further optimize the scene representation through a limited number of iterations, and only use known views for supervision. ### Experimental results - **Quantitative evaluation**: On the DTU and LLFF datasets, FewViewGS performs excellently in terms of PSNR, SSIM, and LPIPS metrics, especially reaching the state - of - the - art level on the LLFF dataset. - **Ablation experiment**: The effectiveness of each component is verified through ablation experiments, especially the crucial role of local - preserving regularization and new - view consistency loss in improving the rendering quality. ### Conclusion The FewViewGS system proposed in the paper performs excellently in the few - sample new - view synthesis task, can effectively solve the over - fitting problem of existing methods under the condition of sparse input images, and provide high - quality rendering results.

FewViewGS: Gaussian Splatting with Few View Matching and Multi-stage Training

MVGS: Multi-view-regulated Gaussian Splatting for Novel View Synthesis

Few-shot Novel View Synthesis using Depth Aware 3D Gaussian Splatting

FSGS: Real-Time Few-shot View Synthesis using Gaussian Splatting

SparseGS: Real-Time 360° Sparse View Synthesis using Gaussian Splatting

Binocular-Guided 3D Gaussian Splatting with View Consistency for Sparse View Synthesis

Structure Consistent Gaussian Splatting with Matching Prior for Few-shot Novel View Synthesis

Self-Ensembling Gaussian Splatting for Few-shot Novel View Synthesis

CoherentGS: Sparse Novel View Synthesis with Coherent 3D Gaussians

MVPGS: Excavating Multi-view Priors for Gaussian Splatting from Sparse Input Views

MCGS: Multiview Consistency Enhancement for Sparse-View 3D Gaussian Radiance Fields

SGD: Street View Synthesis with Gaussian Splatting and Diffusion Prior

NovelGS: Consistent Novel-view Denoising via Large Gaussian Reconstruction Model

Spacetime Gaussian Feature Splatting for Real-Time Dynamic View Synthesis

LoopSparseGS: Loop Based Sparse-View Friendly Gaussian Splatting

Gaussian in the Dark: Real-Time View Synthesis From Inconsistent Dark Images Using Gaussian Splatting

GPS-Gaussian: Generalizable Pixel-wise 3D Gaussian Splatting for Real-time Human Novel View Synthesis

HybridGS: Decoupling Transients and Statics with 2D and 3D Gaussian Splatting

Wild-GS: Real-Time Novel View Synthesis from Unconstrained Photo Collections

DNGaussian: Optimizing Sparse-View 3D Gaussian Radiance Fields with Global-Local Depth Normalization

Generalizable Human Gaussians for Sparse View Synthesis