Abstract:The advancement of generative radiance fields has pushed the boundary of 3D-aware image synthesis. Motivated by the observation that a 3D object should look realistic from multiple viewpoints, these methods introduce a multi-view constraint as regularization to learn valid 3D radiance fields from 2D images. Despite the progress, they often fall short of capturing accurate 3D shapes due to the shape-color ambiguity, limiting their applicability in downstream tasks. In this work, we address this ambiguity by proposing a novel shading-guided generative implicit model that is able to learn a starkly improved shape representation. Our key insight is that an accurate 3D shape should also yield a realistic rendering under different lighting conditions. This multi-lighting constraint is realized by modeling illumination explicitly and performing shading with various lighting conditions. Gradients are derived by feeding the synthesized images to a discriminator. To compensate for the additional computational burden of calculating surface normals, we further devise an efficient volume rendering strategy via surface tracking, reducing the training and inference time by 24% and 48%, respectively. Our experiments on multiple datasets show that the proposed approach achieves photorealistic 3D-aware image synthesis while capturing accurate underlying 3D shapes. We demonstrate improved performance of our approach on 3D shape reconstruction against existing methods, and show its applicability on image relighting. Our code will be released at <a class="link-external link-https" href="https://github.com/XingangPan/ShadeGAN" rel="external noopener nofollow">this https URL</a>.

Shape generation via learning an adaptive multimodal prior

3D-Aware Image Synthesis Via Learning Structural and Textural Representations

Learn, Imagine and Create: Text-to-Image Generation from Prior Knowledge.

Generative Creativity: Adversarial Learning For Bionic Design

Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling

Learning Adversarial 3D Model Generation with 2D Image Enhancer

Synthesizing 3D Shapes From Silhouette Image Collections Using Multi-Projection Generative Adversarial Networks

3DQD: Generalized Deep 3D Shape Prior via Part-Discretized Diffusion Process

XDGAN: Multi-Modal 3D Shape Generation in 2D Space

Shape Inpainting using 3D Generative Adversarial Network and Recurrent Convolutional Networks

Get3DHuman: Lifting StyleGAN-Human into a 3D Generative Model using Pixel-aligned Reconstruction Priors

Michelangelo: Conditional 3D Shape Generation based on Shape-Image-Text Aligned Latent Representation

Few-shot 3D Shape Generation

A Shading-Guided Generative Implicit Model for Shape-Accurate 3D-Aware Image Synthesis

Do 2D GANs Know 3D Shape? Unsupervised 3D shape reconstruction from 2D Image GANs

Unsupervised 3D Shape Completion through GAN Inversion

Non-Parametric Priors For Generative Adversarial Networks

Learning Part Generation and Assembly for Structure-aware Shape Synthesis

A Variational Autoencoder Cascade Generative Adversarial Network for Scalable 3D Object Generation and Reconstruction

Learning Energy-Based 3D Descriptor Networks for Volumetric Shape Synthesis and Analysis

Learning to Generate 3D Shapes from a Single Example