Abstract:We present a real-time facial tracking and animation system based on a Kinect sensor with video and audio input. Our method requires no user-specific training and is robust to occlusions, large head rotations, and background noise. Given the color, depth and speech audio frames captured from an actor, our system first reconstructs 3D facial expressions and 3D mouth shapes from color and depth input with a multi-linear model. Concurrently a speaker-independent DNN acoustic model is applied to extract phoneme state posterior probabilities (PSPP) from the audio frames. After that, a lip motion regressor refines the 3D mouth shape based on both PSPP and expression weights of the 3D mouth shapes, as well as their confidences. Finally, the refined 3D mouth shape is combined with other parts of the 3D face to generate the final result. The whole process is fully automatic and executed in real time. The key component of our system is a data-driven regresor for modeling the correlation between speech data and mouth shapes. Based on a precaptured database of accurate 3D mouth shapes and associated speech audio from one speaker, the regressor jointly uses the input speech and visual features to refine the mouth shape of a new actor. We also present an improved DNN acoustic model. It not only preserves accuracy but also achieves real-time performance. Our method efficiently fuses visual and acoustic information for 3D facial performance capture. It generates more accurate 3D mouth motions than other approaches that are based on audio or video input only. It also supports video or audio only input for real-time facial animation. We evaluate the performance of our system with speech and facial expressions captured from different actors. Results demonstrate the efficiency and robustness of our method.

Acoustic VR in the Mouth: A Real-Time Speech-Driven Visual Tongue System.

A Speech-Driven 3-D Tongue Model with Realistic Movement in Mandarin Chinese.

Progress in animation of an EMA-controlled tongue model for acoustic-visual speech synthesis

Speaker-independent Lips and Tongue Visualization of Vowels

3D Singing Head for Music VR

A Speech-Driven 3-D Lip Synthesis with Realistic Dynamics in Mandarin Chinese

High-fidelity facial and speech animation for VR HMDs

Real-time Ultrasound-enhanced Multimodal Imaging of Tongue using 3D Printable Stabilizer System: A Deep Learning Approach

Synthesizing 3D Trump: Predicting and Visualizing the Relationship Between Text, Speech, and Articulatory Movements.

Visualization of Mandarin articulation by using a physiological articulatory model

Audio to Deep Visual: Speaking Mouth Generation Based on 3D Sparse Landmarks

A mass-spring tongue model with efficient collision detection and response during speech

Combined X-ray and facial videos for phoneme-level articulator dynamics

Speech2rtMRI: Speech-Guided Diffusion Model for Real-time MRI Video of the Vocal Tract during Speech

Contour-based 3d tongue motion visualization using ultrasound image sequences

Speech production of vowel sequences using a physiological articulatory model

An Emotional Text-Driven 3D Visual Pronunciation System for Mandarin Chinese

Development of a 3D tongue motion visualization platform based on ultrasound image sequences

A novel 3D geometric articulatory model

Tongue Shape Synthesis Based on Active Shape Model.

Video-audio Driven Real-Time Facial Animation.