Abstract:Video-driven 3D facial animation transfer aims to drive avatars to reproduce the expressions of actors. Existing methods have achieved remarkable results by constraining both geometric and perceptual consistency. However, geometric constraints (like those designed on facial landmarks) are insufficient to capture subtle emotions, while expression features trained on classification tasks lack fine granularity for complex emotions. To address this, we propose \textbf{FreeAvatar}, a robust facial animation transfer method that relies solely on our learned expression representation. Specifically, FreeAvatar consists of two main components: the expression foundation model and the facial animation transfer model. In the first component, we initially construct a facial feature space through a face reconstruction task and then optimize the expression feature space by exploring the similarities among different expressions. Benefiting from training on the amounts of unlabeled facial images and re-collected expression comparison dataset, our model adapts freely and effectively to any in-the-wild input facial images. In the facial animation transfer component, we propose a novel Expression-driven Multi-avatar Animator, which first maps expressive semantics to the facial control parameters of 3D avatars and then imposes perceptual constraints between the input and output images to maintain expression consistency. To make the entire process differentiable, we employ a trained neural renderer to translate rig parameters into corresponding images. Furthermore, unlike previous methods that require separate decoders for each avatar, we propose a dynamic identity injection module that allows for the joint training of multiple avatars within a single network.

Hierarchical facial expression animation by motion capture data

Video Tracked Facial Expression Animation

Video-driven state-aware facial animation

High Resolution Acquisition, Learning And Transfer Of Dynamic 3-D Facial Expressions

Facial Expression Animation Based on Physical Model

Video-Based Facial Animation with Detailed Appearance Texture

Interactive Modeling of 3D Facial Expressions with Hierarchical Gaussian Process Latent Variable Models

Expressive Face Animation Synthesis Based on Dynamic Mapping Method

Pose-Controllable 3D Facial Animation Synthesis using Hierarchical Audio-Vertex Attention

Interactive Facial-Geometric-Feature Animation For Generating Expressions Of Novel Faces

Data-driven Facial Animation Via Hypergraph Learning

A data-driven approach for facial expression synthesis in video

Subtle Facial Expression Synthesis Using Motion Manifold Embedding and Nonlinear Decomposable Generative Models

FreeAvatar: Robust 3D Facial Animation Transfer by Learning an Expression Foundation Model

Video-Driven Neural Physically-Based Facial Asset for Production

Neuromuscular Control of the Face-Head-Neck Biomechanical Complex With Learning-Based Expression Transfer From Images and Videos

Face Animation Based on Large Audiovisual Database

Facial Expression Retargeting from Human to Avatar Made Easy

A Piece-Wise Learning Approach To 3d Facial Animation

Controllable high-fidelity facial performance transfer

Data-driven facial expression synthesis via Laplacian deformation