Abstract:We present a new system that applies an example-based learning method to learn facial motion patterns from a video sequence of individual facial behavior such as lip motion and facial expressions, and using that to create vivid three-dimensional (3-D) face animation according to the definition of MPEG-4 face animation parameters. The system consists of three key modules, face tracking, pattern learning, and face animation. In face tracking, to reduce the complexity of the tracking process, a novel coarse-to-fine strategy combined with a Kalman filter is proposed for localizing key facial landmarks in each image of the video. The landmarks' sequence is normalized into a visual feature matrix and then fed to the next step of process. In pattern learning, in the pretraining stage, the parameters of the camera that took the video are requested with the training video data so the system can estimate the basic mapping from a normalized two-dimensional (2-D) visual feature matrix to the representation in 3-D MPEG-4 face animation parameter space, in assistance with the computer vision method. In the practice stage, considering that in most cases camera parameters are not provided with video data, the system uses machine learning technology to complement the incomplete 3-D information for the mapping that information is needed in face orientation presentation. The example-based learning in this system integrates several methods including clustering, HMM, and ANN to make a better conversion from a 2-D to 3-D model and better estimation of incomplete 3-D information for good mapping; this will be used to drive face animation thereafter. In face animation, the system can synthesize face animation following any type of face motion in video. Experiments show that our system produces more vivid face motion animation, compared to other early systems.

3D Shape Regression for Real-Time Facial Animation

LBF Based 3D Regression for Facial Animation

Real-time Facial Animation on Mobile Devices

Displaced Dynamic Expression Regression for Real-Time Facial Tracking and Animation

Face Alignment Based on 3D Face Shape Model and Markov Random Field.

A Piece-Wise Learning Approach To 3d Facial Animation

Video Tracked Facial Expression Animation

Real-time non-rigid shape recovery via active appearance models for augmented reality

Automatic facial expression recognition on a single 3D face by exploring shape deformation.

Automatic 3D Face Modeling Using 2D Active Appearance Models

Real-time high-fidelity facial performance capture

Real-time Simultaneous 3D Head Modeling and Facial Motion Capture with an RGB-D camera

Emotion-Preserving Blendshape Update with Real-Time Face Tracking

Real-Time Facial Expression Mapping for High Resolution 3D Meshes

Real-time Facial Animation with Image-Based Dynamic Avatars.

Video-audio Driven Real-Time Facial Animation.

Deep Energies for Estimating Three-Dimensional Facial Pose and Expression

3D Facial Expression Reconstruction using Cascaded Regression

Personalized 3-D facial expression synthesis based on landmark constraint.

Joint Head Pose and Facial Landmark Regression from Depth Images

Learning and Synthesizing Mpeg-4 Compatible 3-D Face Animation from Video Sequence