Abstract:Abstract Recent years have witnessed significant progress in image-based 3D face reconstruction using deep convolutional neural networks. However, current reconstruction methods often perform improperly in self-occluded regions and can lead to inaccurate correspondences between a 2D input image and a 3D face template, hindering use in real applications. To address these problems, we propose a deep shape reconstruction and texture completion network, SRTC-Net, which jointly reconstructs 3D facial geometry and completes texture with correspondences from a single input face image. In SRTC-Net, we leverage the geometric cues from completed 3D texture to reconstruct detailed structures of 3D shapes. The SRTC-Net pipeline has three stages. The first introduces a correspondence network to identify pixel-wise correspondence between the input 2D image and a 3D template model, and transfers the input 2D image to a U - V texture map. Then we complete the invisible and occluded areas in the U - V texture map using an inpainting network. To get the 3D facial geometries, we predict coarse shape ( U - V position maps) from the segmented face from the correspondence network using a shape network, and then refine the 3D coarse shape by regressing the U - V displacement map from the completed U - V texture map in a pixel-to-pixel way. We examine our methods on 3D reconstruction tasks as well as face frontalization and pose invariant face recognition tasks, using both in-the-lab datasets (MICC, MultiPIE) and in-the-wild datasets (CFP). The qualitative and quantitative results demonstrate the effectiveness of our methods on inferring 3D facial geometry and complete texture; they outperform or are comparable to the state-of-the-art.

Identity-preserving 3D Facial Completion under Skull Constraints

Identity Preserving Face Completion for Large Ocular Region Occlusion

FaceCom: Towards High-fidelity 3D Facial Shape Completion via Optimization and Inpainting Guidance

From Skulls to Faces: A Deep Generative Framework for Realistic 3D Craniofacial Reconstruction.

Skull-to-Face: Anatomy-Guided 3D Facial Reconstruction and Editing

Superimposition-guided Facial Reconstruction from Skull

Driving Mutual Advancement of 3D Reconstruction and Inpainting for Masked Faces

End-to-End 3D Facial Shape Reconstruction from an Unconstrained Image.

Joint 3D facial shape reconstruction and texture completion from a single image

Extreme 3D Face Reconstruction: Seeing Through Occlusions

3DFaceFill: An Analysis-By-Synthesis Approach to Face Completion

Dense 3D Face Reconstruction from a Single RGB Image

Face Super-Resolution Guided by 3D Facial Priors

A Hierarchical Dense Deformable Model for 3D Face Reconstruction from Skull.

3D Face Reconstruction and Semantic Annotation from Single Depth Image

3D face reconstruction from skull by regression modeling in shape parameter spaces.

The Craniofacial Reconstruction from the Local Structural Diversity of Skulls

Face Restoration via Plug-and-Play 3D Facial Priors

Face Reconstruction from Skull Based on Least Squares Canonical Dependency Analysis

IPFC: an Attentive Face Completion Network with Identity Preserving

An End-to-End Conditional Generative Adversarial Network Based on Depth Map for 3D Craniofacial Reconstruction