Abstract:High-fidelity face digitization solutions often combine multi-view stereo (MVS) techniques for 3D reconstruction and a non-rigid registration step to establish dense correspondence across identities and expressions. A common problem is the need for manual clean-up after the MVS step, as 3D scans are typically affected by noise and outliers and contain hairy surface regions that need to be cleaned up by artists. Furthermore, mesh registration tends to fail for extreme facial expressions. Most learning-based methods use an underlying 3D morphable model (3DMM) to ensure robustness, but this limits the output accuracy for extreme facial expressions. In addition, the global bottleneck of regression architectures cannot produce meshes that tightly fit the ground truth surfaces. We propose ToFu, Topologically consistent Face from multi-view, a geometry inference framework that can produce topologically consistent meshes across facial identities and expressions using a volumetric representation instead of an explicit underlying 3DMM. Our novel progressive mesh generation network embeds the topological structure of the face in a feature volume, sampled from geometry-aware local features. A coarse-to-fine architecture facilitates dense and accurate facial mesh predictions in a consistent mesh topology. ToFu further captures displacement maps for pore-level geometric details and facilitates high-quality rendering in the form of albedo and specular reflectance maps. These high-quality assets are readily usable by production studios for avatar creation, animation and physically-based skin rendering. We demonstrate state-of-the-art geometric and correspondence accuracy, while only taking 0.385 seconds to compute a mesh with 10K vertices, which is three orders of magnitude faster than traditional techniques. The code and the model are available for research purposes at <a class="link-external link-https" href="https://tianyeli.github.io/tofu" rel="external noopener nofollow">this https URL</a>.

Dynamic 4D facial capture pipeline with appearance driven progressive retopology based on optical flow

High-fidelity 3D Face Reconstruction with Multi-Scale Details

Topo4D: Topology-Preserving Gaussian Splatting for High-Fidelity 4D Head Capture

Progressive 3D Reconstruction of Planar-Faced Manifold Objects with DRF-Based Line Drawing Decomposition

4D facial analysis: A survey of datasets, algorithms and applications

4D Agnostic Real-Time Facial Animation Pipeline for Desktop Scenarios

Video-Based Facial Animation with Detailed Appearance Texture

Topologically Consistent Multi-View Face Inference Using Volumetric Sampling

Dynamic facial asset and rig generation from a single scan

3DFlowRenderer: One-shot Face Re-enactment via Dense 3D Facial Flow Estimation

Light-Weight Multi-view Topology Consistent Facial Geometry and Reflectance Capture.

Skull-to-Face: Anatomy-Guided 3D Facial Reconstruction and Editing

Dynamic 3D Facial Expression Reconstruction from Images

4-D Reconstruction for Dynamic Fluorescence Diffuse Optical Tomography

FaceFolds: Meshed Radiance Manifolds for Efficient Volumetric Rendering of Dynamic Faces

Real-time Facial Animation with Image-Based Dynamic Avatars.

FacialFlowNet: Advancing Facial Optical Flow Estimation with a Diverse Dataset and a Decomposed Model

Video-Driven Neural Physically-Based Facial Asset for Production

A Method of Facial Animation Retargeting Based on Motion Capture

Extreme 3D Face Reconstruction: Seeing Through Occlusions

High Quality Binocular Facial Performance Capture from Partially Blurred Image Sequence