Abstract:3D-aware generative adversarial networks (GANs) synthesize high-fidelity and multi-view-consistent facial images using only collections of single-view 2D imagery. Towards fine-grained control over facial attributes, recent efforts incorporate 3D Morphable Face Model (3DMM) to describe deformation in generative radiance fields either explicitly or implicitly. Explicit methods provide fine-grained expression control but cannot handle topological changes caused by hair and accessories, while implicit ones can model varied topologies but have limited generalization caused by the unconstrained deformation fields. We propose a novel 3D GAN framework for unsupervised learning of generative, high-quality and 3D-consistent facial avatars from unstructured 2D images. To achieve both deformation accuracy and topological flexibility, we propose a 3D representation called Generative Texture-Rasterized Tri-planes. The proposed representation learns Generative Neural Textures on top of parametric mesh templates and then projects them into three orthogonal-viewed feature planes through rasterization, forming a tri-plane feature representation for volume rendering. In this way, we combine both fine-grained expression control of mesh-guided explicit deformation and the flexibility of implicit volumetric representation. We further propose specific modules for modeling mouth interior which is not taken into account by 3DMM. Our method demonstrates state-of-the-art 3D-aware synthesis quality and animation ability through extensive experiments. Furthermore, serving as 3D prior, our animatable 3D representation boosts multiple applications including one-shot facial avatars and 3D-aware stylization. Project page: https://mrtornado24.github.io/Next3D/. Code: https://github.com/MrTornado24/Next3D.

Synthesizing 3D VR Sketch Using Generative Adversarial Neural Network

3D-Aware Image Synthesis Via Learning Structural and Textural Representations

Rapid 3D Model Generation with Intuitive 3D Input

Virtual View Generation Based on 3D-Dense-attentive GAN Networks

High-Fidelity 3D Model Generation with Relightable Appearance from Single Freehand Sketches and Text Guidance

pi-GAN: Periodic Implicit Generative Adversarial Networks for 3D-Aware Image Synthesis

VRCopilot: Authoring 3D Layouts with Generative AI Models in VR

3D VR Sketch Guided 3D Shape Prototyping and Exploration

Next3D: Generative Neural Texture Rasterization for 3D-Aware Head Avatars

Sketch Your Own GAN

Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling

CAD: Photorealistic 3D Generation via Adversarial Distillation

Generative Novel View Synthesis with 3D-Aware Diffusion Models

A Variational Autoencoder Cascade Generative Adversarial Network for Scalable 3D Object Generation and Reconstruction

Generative View Synthesis: From Single-view Semantics to Novel-view Images

PV3D: A 3D Generative Model for Portrait Video Generation

GANHead: Towards Generative Animatable Neural Head Avatars

High-Quality Facial Photo-Sketch Synthesis Using Multi-Adversarial Networks

Face Synthesis from Visual Attributes via Sketch using Conditional VAEs and GANs

Reality3DSketch: Rapid 3D Modeling of Objects from Single Freehand Sketches

A Survey on Deep Generative 3D-aware Image Synthesis