Abstract:We present a real-time facial tracking and animation system based on a Kinect sensor with video and audio input. Our method requires no user-specific training and is robust to occlusions, large head rotations, and background noise. Given the color, depth and speech audio frames captured from an actor, our system first reconstructs 3D facial expressions and 3D mouth shapes from color and depth input with a multi-linear model. Concurrently a speaker-independent DNN acoustic model is applied to extract phoneme state posterior probabilities (PSPP) from the audio frames. After that, a lip motion regressor refines the 3D mouth shape based on both PSPP and expression weights of the 3D mouth shapes, as well as their confidences. Finally, the refined 3D mouth shape is combined with other parts of the 3D face to generate the final result. The whole process is fully automatic and executed in real time. The key component of our system is a data-driven regresor for modeling the correlation between speech data and mouth shapes. Based on a precaptured database of accurate 3D mouth shapes and associated speech audio from one speaker, the regressor jointly uses the input speech and visual features to refine the mouth shape of a new actor. We also present an improved DNN acoustic model. It not only preserves accuracy but also achieves real-time performance. Our method efficiently fuses visual and acoustic information for 3D facial performance capture. It generates more accurate 3D mouth motions than other approaches that are based on audio or video input only. It also supports video or audio only input for real-time facial animation. We evaluate the performance of our system with speech and facial expressions captured from different actors. Results demonstrate the efficiency and robustness of our method.

A real-time speech driven talking avatar based on deep neural network.

Acoustic to Articulatory Mapping with Deep Neural Network

APB2FaceV2: Real-Time Audio-Guided Multi-Face Reenactment

An Online Speech Driven Talking Head System

Video-audio Driven Real-Time Facial Animation.

Live Speech Portraits: Real-Time Photorealistic Talking-Head Animation

Audio-driven Talking Face Video Generation with Natural Head Pose

Real-time Synthesis of Chinese Visual Speech and Facial Expressions Using MPEG-4 FAP Features in a Three-Dimensional Avatar

Text to Avatar in Multi-modal Human Computer Interface

Sem-Avatar: Semantic Controlled Neural Field for High-Fidelity Audio Driven Avatar.

Speech driven photo realistic facial animation based on an articulatory DBN model and AAM features

Audio-driven facial animation by joint end-to-end learning of pose and emotion

Real-time Speech-Driven Animation of Expressive Talking Faces.

Real-time Synthesis of Chinese Visua using MPEG-4 FAP Features in a

Ada-TTA: Towards Adaptive High-Quality Text-to-Talking Avatar Synthesis

Acoustic VR in the Mouth: A Real-Time Speech-Driven Visual Tongue System.

VASA-1: Lifelike Audio-Driven Talking Faces Generated in Real Time

Speech Driven MPEG-4 Based Face Animation via Neural Network

Low Level Descriptors Based DBLSTM Bottleneck Feature for Speech Driven Talking Avatar

TALK-Act: Enhance Textural-Awareness for 2D Speaking Avatar Reenactment with Diffusion Model

Expressive Speech Driven Talking Avatar Synthesis with DBLSTM Using Limited Amount of Emotional Bimodal Data