Abstract:Novel view synthesis from a sparse set of input images is a challenging problem of great practical interest, especially when camera poses are absent or inaccurate. Direct optimization of camera poses and usage of estimated depths in neural radiance field algorithms usually do not produce good results because of the coupling between poses and depths, and inaccuracies in monocular depth estimation. In this paper, we leverage the recent 3D Gaussian splatting method to develop a novel construct-and-optimize method for sparse view synthesis without camera poses. Specifically, we construct a solution progressively by using monocular depth and projecting pixels back into the 3D world. During construction, we optimize the solution by detecting 2D correspondences between training views and the corresponding rendered images. We develop a unified differentiable pipeline for camera registration and adjustment of both camera poses and depths, followed by back-projection. We also introduce a novel notion of an expected surface in Gaussian splatting, which is critical to our optimization. These steps enable a coarse solution, which can then be low-pass filtered and refined using standard optimization methods. We demonstrate results on the Tanks and Temples and Static Hikes datasets with as few as three widely-spaced views, showing significantly better quality than competing methods, including those with approximate camera pose information. Moreover, our results improve with more views and outperform previous InstantNGP and Gaussian Splatting algorithms even when using half the dataset. Project page: <a class="link-external link-https" href="https://raymondjiangkw.github.io/cogs.github.io/" rel="external noopener nofollow">this https URL</a>

SimVS: Simulating World Inconsistencies for Robust View Synthesis

Ivs-Net: Learning Human View Synthesis from Internet Videos

Consistent-1-to-3: Consistent Image to 3D View Synthesis via Geometry-aware Diffusion Models

Stable View Synthesis

Generative View Synthesis: From Single-view Semantics to Novel-view Images

ViewCrafter: Taming Video Diffusion Models for High-fidelity Novel View Synthesis

PixelSynth: Generating a 3D-Consistent Experience from a Single Image

View Synthesis with Multi-scale Cost Aggregation and Confidence Prior

Long-Term Photometric Consistent Novel View Synthesis with Diffusion Models

SV3D: Novel Multi-view Synthesis and 3D Generation from a Single Image using Latent Video Diffusion

Dynamic scene novel view synthesis via deferred spatio-temporal consistency

MOVIS: Enhancing Multi-Object Novel View Synthesis for Indoor Scenes

View Synthesis of Dynamic Scenes based on Deep 3D Mask Volume

ViVid-1-to-3: Novel View Synthesis with Video Diffusion Models

Modeling Ambient Scene Dynamics for Free-view Synthesis

A Construct-Optimize Approach to Sparse View Synthesis without Camera Pose

Deep View Synthesis Via Self-Consistent Generative Network

CompNVS: Novel View Synthesis with Scene Completion

Sat2Vid - Street-view Panoramic Video Synthesis from a Single Satellite Image.

ReconX: Reconstruct Any Scene from Sparse Views with Video Diffusion Model