Abstract:Previous animatable 3D-aware GANs for human generation have primarily focused on either the human head or full body. However, head-only videos are relatively uncommon in real life, and full body generation typically does not deal with facial expression control and still has challenges in generating high-quality results. Towards applicable video avatars, we present an animatable 3D-aware GAN that generates portrait images with controllable facial expression, head pose, and shoulder movements. It is a generative model trained on unstructured 2D image collections without using 3D or video data. For the new task, we base our method on the generative radiance manifold representation and equip it with learnable facial and head-shoulder deformations. A dual-camera rendering and adversarial learning scheme is proposed to improve the quality of the generated faces, which is critical for portrait images. A pose deformation processing network is developed to generate plausible deformations for challenging regions such as long hair. Experiments show that our method, trained on unstructured 2D images, can generate diverse and high-quality 3D portraits with desired control over different properties.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is the application limitations encountered by existing 3D - aware generation methods when generating animatable face or full - body images. Specifically: 1. **Limitations of existing methods**: - **Only generate the head**: Although high - quality facial expressions and hair can be generated, such videos are relatively rare in real life and are therefore less applicable in practical applications. - **Full - body generation**: Although images containing other parts of the body can be generated and the pose can be controlled, challenges still remain in generating high - quality full - body images, especially in terms of facial expression control. 2. **New tasks proposed in the paper**: - The paper proposes a brand - new task, that is, to generate animatable 3D portraits including the head and shoulder areas. This task aims to fill the gaps in practical applications such as video conferencing and virtual hosts in existing methods. 3. **Main contributions**: - **Animatable 3D portrait generation**: For the first time, a 3D - aware Generative Adversarial Network (GAN) capable of generating the head and shoulder areas is proposed, and facial expressions and the poses of the head and shoulders can be controlled. - **Dual - camera rendering and adversarial learning scheme**: A dual - camera rendering and adversarial learning scheme is proposed, which significantly improves the quality of facial generation. - **Pose - deformation processing module**: A pose - deformation processing module is designed to achieve smooth and reasonable hair deformation, solving the discontinuity problem of long hair when the head rotates. Through these innovations, the paper aims to generate diverse and high - quality 3D portrait images while providing flexible control over different attributes, thereby better meeting the needs of practical applications.

AniPortraitGAN: Animatable 3D Portrait Generation from 2D Image Collections

Learning Full-Head 3D GANs from a Single-View Portrait Dataset

3DFaceShop: Explicitly Controllable 3D-Aware Portrait Generation

3DPortraitGAN: Learning One-Quarter Headshot 3D GANs from a Single-View Portrait Dataset with Diverse Body Poses

GANHead: Towards Generative Animatable Neural Head Avatars

PV3D: A 3D Generative Model for Portrait Video Generation

Generating Animatable 3D Cartoon Faces from Single Portraits

AniGAN: Style-Guided Generative Adversarial Networks for Unsupervised Anime Face Generation

GAN-Avatar: Controllable Personalized GAN-based Human Head Avatar

Portrait3D: Text-Guided High-Quality 3D Portrait Generation Using Pyramid Representation and GANs Prior

SofGAN: A Portrait Image Generator with Dynamic Styling

MyPortrait: Morphable Prior-Guided Personalized Portrait Generation

XAGen: 3D Expressive Human Avatars Generation

AgileGAN3D: Few-Shot 3D Portrait Stylization by Augmented Transfer Learning

AniPortrait: Audio-Driven Synthesis of Photorealistic Portrait Animation

PuppeteerGAN: Arbitrary Portrait Animation With Semantic-Aware Appearance Transformation

Two Birds with One Stone: Iteratively Learn Facial Attributes with GANs.

Generative Adversarial Talking Head: Bringing Portraits to Life with a Weakly Supervised Neural Network

Next3D: Generative Neural Texture Rasterization for 3D-Aware Head Avatars

AniArtAvatar: Animatable 3D Art Avatar from a Single Image