Abstract:Existing neural rendering-based text-to-3D-portrait generation methods typically make use of human geometry prior and diffusion models to obtain guidance. However, relying solely on geometry information introduces issues such as the Janus problem, over-saturation, and over-smoothing. We present Portrait3D , a novel neural rendering-based framework with a novel joint geometry-appearance prior to achieve text-to-3D-portrait generation that overcomes the aforementioned issues. To accomplish this, we train a 3D portrait generator, 3DPortraitGAN, as a robust prior. This generator is capable of producing 360° canonical 3D portraits, serving as a starting point for the subsequent diffusion-based generation process. To mitigate the "grid-like" artifact caused by the high-frequency information in the feature-map-based 3D representation commonly used by most 3D-aware GANs, we integrate a novel pyramid tri-grid 3D representation into 3DPortraitGAN. To generate 3D portraits from text, we first project a randomly generated image aligned with the given prompt into the pre-trained 3DPortraitGAN's latent space. The resulting latent code is then used to synthesize a pyramid tri-grid. Beginning with the obtained pyramid tri-grid , we use score distillation sampling to distill the diffusion model's knowledge into the pyramid tri-grid. Following that, we utilize the diffusion model to refine the rendered images of the 3D portrait and then use these refined images as training data to further optimize the pyramid tri-grid , effectively eliminating issues with unrealistic color and unnatural artifacts. Our experimental results show that Portrait3D can produce realistic, high-quality, and canonical 3D portraits that align with the prompt.

HumanGen: Generating Human Radiance Fields with Explicit Priors

3D-Aware Image Synthesis Via Learning Structural and Textural Representations

Get3DHuman: Lifting StyleGAN-Human into a 3D Generative Model using Pixel-aligned Reconstruction Priors

3D-Aware Semantic-Guided Generative Model for Human Synthesis.

VeRi3D: Generative Vertex-based Radiance Fields for 3D Controllable Human Image Synthesis

SemanticHuman-HD: High-Resolution Semantic Disentangled 3D Human Generation

Generative Neural Articulated Radiance Fields

MVHuman: Tailoring 2D Diffusion with Multi-view Sampling For Realistic 3D Human Generation

LumiGAN: Unconditional Generation of Relightable 3D Human Faces

GRAM: Generative Radiance Manifolds for 3D-Aware Image Generation

HumanGaussian: Text-Driven 3D Human Generation with Gaussian Splatting

HDHumans: A Hybrid Approach for High-fidelity Digital Humans

InceptionHuman: Controllable Prompt-to-NeRF for Photorealistic 3D Human Generation

Portrait3D: Text-Guided High-Quality 3D Portrait Generation Using Pyramid Representation and GANs Prior

GRAM-HD: 3D-Consistent Image Generation at High Resolution with Generative Radiance Manifolds

3D Human Reconstruction in the Wild with Synthetic Data Using Generative Models

ReN Human: Learning Relightable Neural Implicit Surfaces for Animatable Human Rendering

High-fidelity Facial Avatar Reconstruction from Monocular Video with Generative Priors

Learning Detailed Radiance Manifolds for High-Fidelity and 3D-Consistent Portrait Synthesis from Monocular Image

EG-HumanNeRF: Efficient Generalizable Human NeRF Utilizing Human Prior for Sparse View