Abstract:While recent works have achieved great success on image-to-3D object generation, high quality and fidelity 3D head generation from a single image remains a great challenge. Previous text-based methods for generating 3D heads were limited by text descriptions and image-based methods struggled to produce high-quality head geometry. To handle this challenging problem, we propose a novel framework, ID-Sculpt, to generate high-quality 3D heads while preserving their identities. Our work incorporates the identity information of the portrait image into three parts: 1) geometry initialization, 2) geometry sculpting, and 3) texture generation stages. Given a reference portrait image, we first align the identity features with text features to realize ID-aware guidance enhancement, which contains the control signals representing the face information. We then use the canny map, ID features of the portrait image, and a pre-trained text-to-normal/depth diffusion model to generate ID-aware geometry supervision, and 3D-GAN inversion is employed to generate ID-aware geometry initialization. Furthermore, with the ability to inject identity information into 3D head generation, we use ID-aware guidance to calculate ID-aware Score Distillation (ISD) for geometry sculpting. For texture generation, we adopt the ID Consistent Texture Inpainting and Refinement which progressively expands the view for texture inpainting to obtain an initialization UV texture map. We then use the ID-aware guidance to provide image-level supervision for noisy multi-view images to obtain a refined texture map. Extensive experiments demonstrate that we can generate high-quality 3D heads with accurate geometry and texture from a single in-the-wild portrait image.

UV-IDM: Identity-Conditioned Latent Diffusion Model for Face UV-Texture Generation

A synthesis method for personalized 3D face reconstruction

Weakly-Supervised Photo-realistic Texture Generation for 3D Face Reconstruction

LWSFACE: Light-Weighted Weakly-Supervised 3D Facial Texture Reconstruction From a Single Image.

UVMap-ID: A Controllable and Personalized UV Map Generative Model

IFaceUV: Intuitive Motion Facial Image Generation by Identity Preservation via UV map

Full Face-and-Head 3D Model with Photorealistic Texture

FFHQ-UV: Normalized Facial UV-Texture Dataset for 3D Face Reconstruction.

Self-Supervised 3d Face Reconstruction Based On Multi-View Uv Fusion

A 3D Face Model for Pose and Illumination Invariant Face Recognition

Towards high-fidelity facial UV map generation in real-world

High-Fidelity Facial Albedo Estimation via Texture Quantization

VGG-Tex: A Vivid Geometry-Guided Facial Texture Estimation Model for High Fidelity Monocular 3D Face Reconstruction

Relightify: Relightable 3D Faces from a Single Image via Diffusion Models

UV-GAN: Adversarial Facial UV Map Completion for Pose-invariant Face Recognition

Robust and Detailed Reconstuction of Human Face from Single Image

Fine-Scale Face Fitting and Texture Fusion With Inverse Renderer

ID-Sculpt: ID-aware 3D Head Generation from Single In-the-wild Portrait Image

UltrAvatar: A Realistic Animatable 3D Avatar Diffusion Model with Authenticity Guided Textures

Full Face Texture Generation of Virtual Human