Abstract:Emerging Metaverse applications demand accessible, accurate, and easy-to-use tools for 3D digital human creations in order to depict different cultures and societies as if in the physical world. Recent large-scale vision-language advances pave the way to for novices to conveniently customize 3D content. However, the generated CG-friendly assets still cannot represent the desired facial traits for human characteristics. In this paper, we present DreamFace, a progressive scheme to generate personalized 3D faces under text guidance. It enables layman users to naturally customize 3D facial assets that are compatible with CG pipelines, with desired shapes, textures, and fine-grained animation capabilities. From a text input to describe the facial traits, we first introduce a coarse-to-fine scheme to generate the neutral facial geometry with a unified topology. We employ a selection strategy in the CLIP embedding space, and subsequently optimize both the details displacements and normals using Score Distillation Sampling from generic Latent Diffusion Model. Then, for neutral appearance generation, we introduce a dual-path mechanism, which combines the generic LDM with a novel texture LDM to ensure both the diversity and textural specification in the UV space. We also employ a two-stage optimization to perform SDS in both the latent and image spaces to significantly provides compact priors for fine-grained synthesis. Our generated neutral assets naturally support blendshapes-based facial animations. We further improve the animation ability with personalized deformation characteristics by learning the universal expression prior using the cross-identity hypernetwork. Notably, DreamFace can generate of realistic 3D facial assets with physically-based rendering quality and rich animation ability from video footage, even for fashion icons or exotic characters in cartoons and fiction movies.

Full Face Texture Generation of Virtual Human

GANtlitz: Ultra High Resolution Generative Model for Multi‐Modal Face Textures

Automatic 3D Face Modeling Using 2D Active Appearance Models

Text-Driven Diverse Facial Texture Generation via Progressive Latent-Space Refinement

Efficient Generative Face Completion with Perceptual Deblurring

Towards High-Fidelity Text-Guided 3D Face Generation and Manipulation Using only Images

Towards High-Fidelity 3D Face Reconstruction from In-the-Wild Images Using Graph Convolutional Networks

Towards high-fidelity facial UV map generation in real-world

FaceLit: Neural 3D Relightable Faces

VGG-Tex: A Vivid Geometry-Guided Facial Texture Estimation Model for High Fidelity Monocular 3D Face Reconstruction

Single Image, Any Face: Generalisable 3D Face Generation

Towards Realistic Generative 3D Face Models

UV-GAN: Adversarial Facial UV Map Completion for Pose-invariant Face Recognition

A face template: Improving the face generation quality of multi-stage generative adversarial networks using coarse-grained facial priors

DreamFace: Progressive Generation of Animatable 3D Faces under Text Guidance

Controllable 3D Face Generation with Conditional Style Code Diffusion

3DFaceFill: An Analysis-By-Synthesis Approach to Face Completion

Fast-GANFIT: Generative Adversarial Network for High Fidelity 3D Face Reconstruction

Face Manifold: Manifold Learning for Synthetic Face Generation

AvatarMe++: Facial Shape and BRDF Inference With Photorealistic Rendering-Aware GANs

GANFIT: Generative Adversarial Network Fitting for High Fidelity 3D Face Reconstruction