Abstract:Existing 3D-from-2D generators are typically designed for well-curated single-category datasets, where all the objects have (approximately) the same scale, 3D location, and orientation, and the camera always points to the center of the scene. This makes them inapplicable to diverse, in-the-wild datasets of non-alignable scenes rendered from arbitrary camera poses. In this work, we develop a 3D generator with Generic Priors (3DGP): a 3D synthesis framework with more general assumptions about the training data, and show that it scales to very challenging datasets, like ImageNet. Our model is based on three new ideas. First, we incorporate an inaccurate off-the-shelf depth estimator into 3D GAN training via a special depth adaptation module to handle the imprecision. Then, we create a flexible camera model and a regularization strategy for it to learn its distribution parameters during training. Finally, we extend the recent ideas of transferring knowledge from pre-trained classifiers into GANs for patch-wise trained models by employing a simple distillation-based technique on top of the discriminator. It achieves more stable training than the existing methods and speeds up the convergence by at least 40%. We explore our model on four datasets: SDIP Dogs 256x256, SDIP Elephants 256x256, LSUN Horses 256x256, and ImageNet 256x256, and demonstrate that 3DGP outperforms the recent state-of-the-art in terms of both texture and geometry quality. Code and visualizations: <a class="link-external link-https" href="https://snap-research.github.io/3dgp" rel="external noopener nofollow">this https URL</a>.

pi-GAN: Periodic Implicit Generative Adversarial Networks for 3D-Aware Image Synthesis

3D-Aware Image Synthesis Via Learning Structural and Textural Representations

SDF-based 3D Generative Adversarial Networks for Image and Geometry Generation

SpatialGAN: Progressive Image Generation Based on Spatial Recursive Adversarial Expansion

Multi-View Consistent Generative Adversarial Networks for 3D-Aware Image Synthesis

Latent-Space Laplacian Pyramids for Adversarial Representation Learning with 3D Point Clouds

CIPS-3D: A 3D-Aware Generator of GANs Based on Conditionally-Independent Pixel Synthesis

3D generation on ImageNet

Synthesizing 3D VR Sketch Using Generative Adversarial Neural Network

GRAM: Generative Radiance Manifolds for 3D-Aware Image Generation

3inGAN: Learning a 3D Generative Model from Images of a Self-similar Scene

Generative Adversarial Networks for Image and Video Synthesis: Algorithms and Applications

High-resolution medical image synthesis using progressively grown generative adversarial networks

Mimic3D: Thriving 3D-Aware GANs via 3D-to-2D Imitation

A Survey on Deep Generative 3D-aware Image Synthesis

What You See is What You GAN: Rendering Every Pixel for High-Fidelity Geometry in 3D GANs

A brief study of generative adversarial networks and their applications in image synthesis

Multi3D: 3D-Aware Multimodal Image Synthesis

Towards Unsupervised Learning of Generative Models for 3D Controllable Image Synthesis

Exemplar-based Pattern Synthesis with Implicit Periodic Field Network

GRAM-HD: 3D-Consistent Image Generation at High Resolution with Generative Radiance Manifolds