Abstract:Production-level workflows for producing convincing 3D dynamic human faces have long relied on an assortment of labor-intensive tools for geometry and texture generation, motion capture and rigging, and expression synthesis. Recent neural approaches automate individual components but the corresponding latent representations cannot provide artists with explicit controls as in conventional tools. In this paper, we present a new learning-based, video-driven approach for generating dynamic facial geometries with high-quality physically-based assets. For data collection, we construct a hybrid multiview-photometric capture stage, coupling with ultra-fast video cameras to obtain raw 3D facial assets. We then set out to model the facial expression, geometry and physically-based textures using separate VAEs where we impose a global MLP based expression mapping across the latent spaces of respective networks, to preserve characteristics across respective attributes. We also model the delta information as wrinkle maps for the physically-based textures, achieving high-quality 4K dynamic textures. We demonstrate our approach in high-fidelity performer-specific facial capture and cross-identity facial motion retargeting. In addition, our multi-VAE-based neural asset, along with the fast adaptation schemes, can also be deployed to handle in-the-wild videos. Besides, we motivate the utility of our explicit facial disentangling strategy by providing various promising physically-based editing results with high realism. Comprehensive experiments show that our technique provides higher accuracy and visual fidelity than previous video-driven facial reconstruction and animation methods.

Data-driven Facial Animation Via Hypergraph Learning

Performance-Driven Animation of Hand-Drawn Cartoon Faces

Facial Expression Animation Based on Physical Model

Hierarchical facial expression animation by motion capture data

Video Tracked Facial Expression Animation

Video-driven state-aware facial animation

Animating Cartoon Faces by Multi-View Drawings

High Resolution Acquisition, Learning And Transfer Of Dynamic 3-D Facial Expressions

A Piece-Wise Learning Approach To 3d Facial Animation

Learning a Generalized Physical Face Model From Data

Data-Driven Approach to Synthesizing Facial Animation Using Motion Capture

Interactive Facial-Geometric-Feature Animation For Generating Expressions Of Novel Faces

Example-based performance driven facial shape animation

Pose-Controllable 3D Facial Animation Synthesis using Hierarchical Audio-Vertex Attention

Video-Driven Neural Physically-Based Facial Asset for Production

A data-driven approach for facial expression synthesis in video

Deep Energies for Estimating Three-Dimensional Facial Pose and Expression

Data-driven facial expression synthesis via Laplacian deformation

Neuromuscular Control of the Face-Head-Neck Biomechanical Complex With Learning-Based Expression Transfer From Images and Videos

Expressive Face Animation Synthesis Based on Dynamic Mapping Method