Abstract:Production-level workflows for producing convincing 3D dynamic human faces have long relied on an assortment of labor-intensive tools for geometry and texture generation, motion capture and rigging, and expression synthesis. Recent neural approaches automate individual components but the corresponding latent representations cannot provide artists with explicit controls as in conventional tools. In this paper, we present a new learning-based, video-driven approach for generating dynamic facial geometries with high-quality physically-based assets. For data collection, we construct a hybrid multiview-photometric capture stage, coupling with ultra-fast video cameras to obtain raw 3D facial assets. We then set out to model the facial expression, geometry and physically-based textures using separate VAEs where we impose a global MLP based expression mapping across the latent spaces of respective networks, to preserve characteristics across respective attributes. We also model the delta information as wrinkle maps for the physically-based textures, achieving high-quality 4K dynamic textures. We demonstrate our approach in high-fidelity performer-specific facial capture and cross-identity facial motion retargeting. In addition, our multi-VAE-based neural asset, along with the fast adaptation schemes, can also be deployed to handle in-the-wild videos. Besides, we motivate the utility of our explicit facial disentangling strategy by providing various promising physically-based editing results with high realism. Comprehensive experiments show that our technique provides higher accuracy and visual fidelity than previous video-driven facial reconstruction and animation methods.

Neural Animation and Reenactment of Human Actor Videos

Neural Rendering and Reenactment of Human Actor Videos

Ivs-Net: Learning Human View Synthesis from Internet Videos

Learning Dynamic Textures for Neural Rendering of Human Actors

ReN Human: Learning Relightable Neural Implicit Surfaces for Animatable Human Rendering

Video-Driven Animation of Neural Head Avatars

Neural Novel Actor: Learning a Generalized Animatable Neural Representation for Human Actors.

Neural Capture of Animatable 3D Human from Monocular Video.

Animatable Implicit Neural Representations for Creating Realistic Avatars from Videos

Relightable and Animatable Neural Avatar from Sparse-View Video

Relightable and Animatable Neural Avatars from Videos

Neural Voice Puppetry: Audio-driven Facial Reenactment

Video-Driven Neural Physically-Based Facial Asset for Production

High-Fidelity Neural Human Motion Transfer from Monocular Video

HumanNeRF: Efficiently Generated Human Radiance Field from Sparse Inputs

Video-based Characters

MonoHuman: Animatable Human Neural Field from Monocular Video

Neural Sign Reenactor: Deep Photorealistic Sign Language Retargeting

Learning Neural Volumetric Representations of Dynamic Humans in Minutes.

HR Human: Modeling Human Avatars with Triangular Mesh and High-Resolution Textures from Videos

Neural Relighting and Expression Transfer On Video Portraits