Abstract:Synthesizing novel view images from a few views is a challenging but practical problem. Existing methods often struggle with producing high-quality results or necessitate per-object optimization in such few-view settings due to the insufficient information provided. In this work, we explore leveraging the strong 2D priors in pre-trained diffusion models for synthesizing novel view images. 2D diffusion models, nevertheless, lack 3D awareness, leading to distorted image synthesis and compromising the identity. To address these problems, we propose DreamSparse, a framework that enables the frozen pre-trained diffusion model to generate geometry and identity-consistent novel view image. Specifically, DreamSparse incorporates a geometry module designed to capture 3D features from sparse views as a 3D prior. Subsequently, a spatial guidance model is introduced to convert these 3D feature maps into spatial information for the generative process. This information is then used to guide the pre-trained diffusion model, enabling it to generate geometrically consistent images without tuning it. Leveraging the strong image priors in the pre-trained diffusion models, DreamSparse is capable of synthesizing high-quality novel views for both object and scene-level images and generalising to open-set images. Experimental results demonstrate that our framework can effectively synthesize novel view images from sparse views and outperforms baselines in both trained and open-set category images. More results can be found on our project page: <a class="link-external link-https" href="https://sites.google.com/view/dreamsparse-webpage" rel="external noopener nofollow">this https URL</a>.

Efficient 3D View Synthesis from Single-Image Utilizing Diffusion Priors

Sparse3D: Distilling Multiview-Consistent Diffusion for Object Reconstruction from Sparse Views

3D-free meets 3D priors: Novel View Synthesis from a Single Image with Pretrained Diffusion Guidance

Diffusion Priors for Dynamic View Synthesis from Monocular Videos

SparseFusion: Distilling View-conditioned Diffusion for 3D Reconstruction

Make-It-3D: High-Fidelity 3D Creation from A Single Image with Diffusion Prior

DreamSparse: Escaping from Plato's Cave with 2D Frozen Diffusion Model Given Sparse Views

Novel 3D-Aware Composition Images Synthesis for Object Display with Diffusion Model.

Efficient-3DiM: Learning a Generalizable Single-image Novel-view Synthesizer in One Day

How to Use Diffusion Priors under Sparse Views?

Deceptive-NeRF/3DGS: Diffusion-Generated Pseudo-Observations for High-Quality Sparse-View Reconstruction

Pragmatist: Multiview Conditional Diffusion Models for High-Fidelity 3D Reconstruction from Unposed Sparse Views

ViewCrafter: Taming Video Diffusion Models for High-fidelity Novel View Synthesis

MVDiffusion++: A Dense High-resolution Multi-view Diffusion Model for Single or Sparse-view 3D Object Reconstruction

NerfDiff: Single-image View Synthesis with NeRF-guided Distillation from 3D-aware Diffusion

HiFi-123: Towards High-fidelity One Image to 3D Content Generation

Novel View Synthesis Using Feature-Preserving Depth Map Resampling

Envision3D: One Image to 3D with Anchor Views Interpolation

ReconFusion: 3D Reconstruction with Diffusion Priors

Generative Novel View Synthesis with 3D-Aware Diffusion Models

Single-view 3D Scene Reconstruction with High-fidelity Shape and Texture