Abstract:We present a real-time facial tracking and animation system based on a Kinect sensor with video and audio input. Our method requires no user-specific training and is robust to occlusions, large head rotations, and background noise. Given the color, depth and speech audio frames captured from an actor, our system first reconstructs 3D facial expressions and 3D mouth shapes from color and depth input with a multi-linear model. Concurrently a speaker-independent DNN acoustic model is applied to extract phoneme state posterior probabilities (PSPP) from the audio frames. After that, a lip motion regressor refines the 3D mouth shape based on both PSPP and expression weights of the 3D mouth shapes, as well as their confidences. Finally, the refined 3D mouth shape is combined with other parts of the 3D face to generate the final result. The whole process is fully automatic and executed in real time. The key component of our system is a data-driven regresor for modeling the correlation between speech data and mouth shapes. Based on a precaptured database of accurate 3D mouth shapes and associated speech audio from one speaker, the regressor jointly uses the input speech and visual features to refine the mouth shape of a new actor. We also present an improved DNN acoustic model. It not only preserves accuracy but also achieves real-time performance. Our method efficiently fuses visual and acoustic information for 3D facial performance capture. It generates more accurate 3D mouth motions than other approaches that are based on audio or video input only. It also supports video or audio only input for real-time facial animation. We evaluate the performance of our system with speech and facial expressions captured from different actors. Results demonstrate the efficiency and robustness of our method.

Speech Driven Facial Animation Using Chinese Mandarin Pronunciation Rules

Facial Animation System for Embedded Application

Text-driven Visual Prosody Generation for Embodied Conversational Agents

An Online Speech Driven Talking Head System

Real-time Speech-Driven Animation of Expressive Talking Faces.

A Speech-Driven 3-D Lip Synthesis with Realistic Dynamics in Mandarin Chinese

Real-time Synthesis of Chinese Visua using MPEG-4 FAP Features in a

3D Facial Animation from Chinese Text.

Real-time Synthesis of Chinese Visual Speech and Facial Expressions Using MPEG-4 FAP Features in a Three-Dimensional Avatar

Speech Driven MPEG-4 Based Face Animation via Neural Network

Speech-Driven 3D Face Animation with Composite and Regional Facial Movements

Speech Driven Face Animation based on Dynamic Concatenation Model

Animating a Chinese interactive virtual character

Realistic Visual Speech Synthesis Based on Hybrid Concatenation Method

Face Animation Based on Large Audiovisual Database

Emotional Chinese talking head system

Video-audio Driven Real-Time Facial Animation.

Data Mining and Speech Driven Face Animation

KMTalk: Speech-Driven 3D Facial Animation with Key Motion Embedding

Expressive Face Animation Synthesis Based on Dynamic Mapping Method

Speaker Independent and Multilingual/Mixlingual Speech-Driven Talking Head Generation Using Phonetic Posteriorgrams