Abstract:Production-level workflows for producing convincing 3D dynamic human faces have long relied on an assortment of labor-intensive tools for geometry and texture generation, motion capture and rigging, and expression synthesis. Recent neural approaches automate individual components but the corresponding latent representations cannot provide artists with explicit controls as in conventional tools. In this paper, we present a new learning-based, video-driven approach for generating dynamic facial geometries with high-quality physically-based assets. For data collection, we construct a hybrid multiview-photometric capture stage, coupling with ultra-fast video cameras to obtain raw 3D facial assets. We then set out to model the facial expression, geometry and physically-based textures using separate VAEs where we impose a global MLP based expression mapping across the latent spaces of respective networks, to preserve characteristics across respective attributes. We also model the delta information as wrinkle maps for the physically-based textures, achieving high-quality 4K dynamic textures. We demonstrate our approach in high-fidelity performer-specific facial capture and cross-identity facial motion retargeting. In addition, our multi-VAE-based neural asset, along with the fast adaptation schemes, can also be deployed to handle in-the-wild videos. Besides, we motivate the utility of our explicit facial disentangling strategy by providing various promising physically-based editing results with high realism. Comprehensive experiments show that our technique provides higher accuracy and visual fidelity than previous video-driven facial reconstruction and animation methods.

Implementation of Facial Behavior for Autonomous Nonplayer Characters

Real-Time Audio-Guided Multi-Face Reenactment

APB2FACE: Audio-Guided Face Reenactment with Auxiliary Pose and Blink Signals.

Video-driven state-aware facial animation

Implementation of Continuous Facial Action of Agent Based on Motion Planning

Quick Customization of Particular Human Face and Muscle-Driven Expression Animation

Interactive Animation of Virtual Characters: Application to Virtual Kung-Fu Fighting

Physically-Based Facial Modeling and Animation with Unity3D Game Engine

Facial Expression Animation Based on Physical Model

Implementation of Emotional Facial Actions for Human-Like Agents in a Multi-Tasking Environment

Research and Implement of Facial Action on Human-like Agent

AgentAvatar: Disentangling Planning, Driving and Rendering for Photorealistic Avatar Agents

Research on Facial Action Model for Human-like Agent

Controllable automatic generation of non‐player characters in 3D anime style

A Facial Expression Model For Human-Like Agent

Video-Driven Neural Physically-Based Facial Asset for Production

Personalized Multi-View Face Animation with Lifelike Textures

Emotional states control for on-line game avatars

Head and Facial Gestures Synthesis Using PAD Model for an Expressive Talking Avatar

Synthesis of Combined Facial Expressions Using Anatomy-based Model

Human Expressions Interaction Between Avatar and Virtual World