Abstract:Modern learning-based approaches to 3D-aware image synthesis achieve high photorealism and 3D-consistent viewpoint changes for the generated images. Existing approaches represent instances in a shared canonical space. However, for in-the-wild datasets a shared canonical system can be difficult to define or might not even exist. In this work, we instead model instances in view space, alleviating the need for posed images and learned camera distributions. We find that in this setting, existing GAN-based methods are prone to generating flat geometry and struggle with distribution coverage. We hence propose WildFusion, a new approach to 3D-aware image synthesis based on latent diffusion models (LDMs). We first train an autoencoder that infers a compressed latent representation, which additionally captures the images' underlying 3D structure and enables not only reconstruction but also novel view synthesis. To learn a faithful 3D representation, we leverage cues from monocular depth prediction. Then, we train a diffusion model in the 3D-aware latent space, thereby enabling synthesis of high-quality 3D-consistent image samples, outperforming recent state-of-the-art GAN-based methods. Importantly, our 3D-aware LDM is trained without any direct supervision from multiview images or 3D geometry and does not require posed images or learned pose or camera distributions. It directly learns a 3D representation without relying on canonical camera coordinates. This opens up promising research avenues for scalable 3D-aware image synthesis and 3D content creation from in-the-wild image data. See <a class="link-external link-https" href="https://katjaschwarz.github.io/wildfusion" rel="external noopener nofollow">this https URL</a> for videos of our 3D results.

ViewFusion: Learning Composable Diffusion Models for Novel View Synthesis

ViewFusion: Towards Multi-View Consistency via Interpolated Denoising

Generative Novel View Synthesis with 3D-Aware Diffusion Models

Novel View Synthesis with Pixel-Space Diffusion Models

ViVid-1-to-3: Novel View Synthesis with Video Diffusion Models

Diffusion Priors for Dynamic View Synthesis from Monocular Videos

Efficient-3DiM: Learning a Generalizable Single-image Novel-view Synthesizer in One Day

Light Field Diffusion for Single-View Novel View Synthesis

Deceptive-NeRF/3DGS: Diffusion-Generated Pseudo-Observations for High-Quality Sparse-View Reconstruction

SparseFusion: Distilling View-conditioned Diffusion for 3D Reconstruction

ViewCrafter: Taming Video Diffusion Models for High-fidelity Novel View Synthesis

Long-Term Photometric Consistent Novel View Synthesis with Diffusion Models

Efficient 3D View Synthesis from Single-Image Utilizing Diffusion Priors

MultiDiff: Consistent Novel View Synthesis from a Single Image

VFusion3D: Learning Scalable 3D Generative Models from Video Diffusion Models

WildFusion: Learning 3D-Aware Latent Diffusion Models in View Space

NVS-Solver: Video Diffusion Model as Zero-Shot Novel View Synthesizer

SV4D: Dynamic 3D Content Generation with Multi-Frame and Multi-View Consistency

NerfDiff: Single-image View Synthesis with NeRF-guided Distillation from 3D-aware Diffusion

Re-Nerfing: Improving Novel View Synthesis through Novel View Synthesis

DreamSparse: Escaping from Plato's Cave with 2D Frozen Diffusion Model Given Sparse Views