Abstract:Existing neural rendering-based text-to-3D-portrait generation methods typically make use of human geometry prior and diffusion models to obtain guidance. However, relying solely on geometry information introduces issues such as the Janus problem, over-saturation, and over-smoothing. We present Portrait3D , a novel neural rendering-based framework with a novel joint geometry-appearance prior to achieve text-to-3D-portrait generation that overcomes the aforementioned issues. To accomplish this, we train a 3D portrait generator, 3DPortraitGAN, as a robust prior. This generator is capable of producing 360° canonical 3D portraits, serving as a starting point for the subsequent diffusion-based generation process. To mitigate the "grid-like" artifact caused by the high-frequency information in the feature-map-based 3D representation commonly used by most 3D-aware GANs, we integrate a novel pyramid tri-grid 3D representation into 3DPortraitGAN. To generate 3D portraits from text, we first project a randomly generated image aligned with the given prompt into the pre-trained 3DPortraitGAN's latent space. The resulting latent code is then used to synthesize a pyramid tri-grid. Beginning with the obtained pyramid tri-grid , we use score distillation sampling to distill the diffusion model's knowledge into the pyramid tri-grid. Following that, we utilize the diffusion model to refine the rendered images of the 3D portrait and then use these refined images as training data to further optimize the pyramid tri-grid , effectively eliminating issues with unrealistic color and unnatural artifacts. Our experimental results show that Portrait3D can produce realistic, high-quality, and canonical 3D portraits that align with the prompt.

HiStyle: Reinventing Historic Portraits Via 3D Generative Model

HyperStyle3D: Text-Guided 3D Portrait Stylization via Hypernetworks

Style Fader Generative Adversarial Networks for Style Degree Controllable Artistic Style Transfer

Creative and Diverse Artwork Generation Using Adversarial Networks

Diverse Image Style Transfer Via Invertible Cross-Space Mapping

AgileGAN3D: Few-Shot 3D Portrait Stylization by Augmented Transfer Learning

Few-shots Portrait Generation with Style Enhancement and Identity Preservation

Style Image Harmonization Via Global-Local Style Mutual Guided

Pastiche Master: Exemplar-Based High-Resolution Portrait Style Transfer

A study of style migration generation of traditional Chinese portraits based on DualStyleGAN

Generating Artistic Portraits from Face Photos with Feature Disentanglement and Reconstruction

Fast 3D Stylized Gaussian Portrait Generation From a Single Image With Style Aligned Sampling Loss

Get3DHuman: Lifting StyleGAN-Human into a 3D Generative Model using Pixel-aligned Reconstruction Priors

Pose with Style: Detail-Preserving Pose-Guided Image Synthesis with Conditional StyleGAN

ZePo: Zero-Shot Portrait Stylization with Faster Sampling

Freestyle 3D-Aware Portrait Synthesis Based on Compositional Generative Priors

StyleAvatar3D: Leveraging Image-Text Diffusion Models for High-Fidelity 3D Avatar Generation

Portrait3D: Text-Guided High-Quality 3D Portrait Generation Using Pyramid Representation and GANs Prior

3Dtoonify: Creating Your High-Fidelity 3D Stylized Avatar Easily from 2D Portrait Images

Enhancing the Authenticity of Rendered Portraits with Identity-Consistent Transfer Learning