Abstract:High-fidelity face digitization solutions often combine multi-view stereo (MVS) techniques for 3D reconstruction and a non-rigid registration step to establish dense correspondence across identities and expressions. A common problem is the need for manual clean-up after the MVS step, as 3D scans are typically affected by noise and outliers and contain hairy surface regions that need to be cleaned up by artists. Furthermore, mesh registration tends to fail for extreme facial expressions. Most learning-based methods use an underlying 3D morphable model (3DMM) to ensure robustness, but this limits the output accuracy for extreme facial expressions. In addition, the global bottleneck of regression architectures cannot produce meshes that tightly fit the ground truth surfaces. We propose ToFu, Topologically consistent Face from multi-view, a geometry inference framework that can produce topologically consistent meshes across facial identities and expressions using a volumetric representation instead of an explicit underlying 3DMM. Our novel progressive mesh generation network embeds the topological structure of the face in a feature volume, sampled from geometry-aware local features. A coarse-to-fine architecture facilitates dense and accurate facial mesh predictions in a consistent mesh topology. ToFu further captures displacement maps for pore-level geometric details and facilitates high-quality rendering in the form of albedo and specular reflectance maps. These high-quality assets are readily usable by production studios for avatar creation, animation and physically-based skin rendering. We demonstrate state-of-the-art geometric and correspondence accuracy, while only taking 0.385 seconds to compute a mesh with 10K vertices, which is three orders of magnitude faster than traditional techniques. The code and the model are available for research purposes at <a class="link-external link-https" href="https://tianyeli.github.io/tofu" rel="external noopener nofollow">this https URL</a>.

An Automatic Method of Building 3D Morphable Face Model

A synthesis method for personalized 3D face reconstruction

Automatic 3D Face Modeling Using 2D Active Appearance Models

Building a 3d Morphable Face Model by Using Thin Plate Splines for Face Reconstruction

Face Alignment Based on 3D Face Shape Model and Markov Random Field.

Fast face modeling based on features and flexible mesh model

A Modeling Method for the Human Body Model with Facial Morphology

Automatic facial expression recognition on a single 3D face by exploring shape deformation.

3D face registration by depth-based template matching and active appearance model

Video Tracked Facial Expression Animation

A Deformation Model to Reduce the Effect of Expressions in 3D Face Recognition

3D3M: 3D Modulated Morphable Model for Monocular Face Reconstruction

Automatic 3D Face Modeling from Video

Automatic landmark annotation and dense correspondence registration for 3D human facial images

Topologically Consistent Multi-View Face Inference Using Volumetric Sampling

A Fast 3d Face Reconstruction Method From A Single Image Using Adjustable Model

Automatic Face Modeling and Synthesis Based on Image Pairs

SR-VFA: Accurate Self-Refined Face Alignment in Videos.

Realistic 3D Face Modeling by Fusing Multiple 2D Images

3-D Face Structure Extraction Using Shape Matching Morphing Model

An Automatic MPEG-4 Based Arbitrary Topology 3D Facial Animation System