Portrait3D: Text-Guided High-Quality 3D Portrait Generation Using Pyramid Representation and GANs Prior

Yiqian Wu,Hao Xu,Xiangjun Tang,Xien Chen,Siyu Tang,Zhebin Zhang,Chen Li,Xiaogang Jin

2024-04-16

Abstract:Existing neural rendering-based text-to-3D-portrait generation methods typically make use of human geometry prior and diffusion models to obtain guidance. However, relying solely on geometry information introduces issues such as the Janus problem, over-saturation, and over-smoothing. We present Portrait3D, a novel neural rendering-based framework with a novel joint geometry-appearance prior to achieve text-to-3D-portrait generation that overcomes the aforementioned issues. To accomplish this, we train a 3D portrait generator, 3DPortraitGAN-Pyramid, as a robust prior. This generator is capable of producing 360° canonical 3D portraits, serving as a starting point for the subsequent diffusion-based generation process. To mitigate the "grid-like" artifact caused by the high-frequency information in the feature-map-based 3D representation commonly used by most 3D-aware GANs, we integrate a novel pyramid tri-grid 3D representation into 3DPortraitGAN-Pyramid. To generate 3D portraits from text, we first project a randomly generated image aligned with the given prompt into the pre-trained 3DPortraitGAN-Pyramid's latent space. The resulting latent code is then used to synthesize a pyramid tri-grid. Beginning with the obtained pyramid tri-grid, we use score distillation sampling to distill the diffusion model's knowledge into the pyramid tri-grid. Following that, we utilize the diffusion model to refine the rendered images of the 3D portrait and then use these refined images as training data to further optimize the pyramid tri-grid, effectively eliminating issues with unrealistic color and unnatural artifacts. Our experimental results show that Portrait3D can produce realistic, high-quality, and canonical 3D portraits that align with the prompt.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

### What problem does this paper attempt to solve?

This paper aims to address the problem of generating high-quality 3D portraits based on text and to overcome some limitations of existing methods. Specifically:

1. **Problems with existing methods**:
- Current neural rendering methods typically rely on human geometry priors and diffusion models for guidance.
- Solely relying on geometric information can lead to issues such as the Janus problem, over-saturation, and over-smoothing.

2. **Proposed method**:
- A new neural rendering framework, **Portrait3D**, is proposed, which utilizes a 3D-aware Generative Adversarial Network (GAN) and diffusion model guidance to generate high-quality 3D portraits that match the input text description.
- To overcome the "grid-like" artifacts in feature map representations, a new pyramid tri-grid representation method is proposed.
- Score distillation sampling is used to distill the knowledge of the diffusion model into the pyramid tri-grid, resulting in more realistic 3D portraits.

3. **Main contributions**:
- A new 3D portrait generator, **3DPortraitGAN**, is proposed, which uses the pyramid tri-grid representation to mitigate "grid-like" artifacts and learns the distribution of 360-degree standard 3D portraits as a strong prior for the framework.
- A text-to-3D portrait generation framework, **Portrait3D**, is proposed, which leverages the prior knowledge of the 3D portrait generator and the guidance of the diffusion model to generate high-quality, view-consistent 3D portraits that match the input text prompts.

Through these methods, the paper demonstrates that **Portrait3D** can generate high-quality, realistic 3D portraits that are highly consistent with the input text.