Abstract:We present a real-time facial tracking and animation system based on a Kinect sensor with video and audio input. Our method requires no user-specific training and is robust to occlusions, large head rotations, and background noise. Given the color, depth and speech audio frames captured from an actor, our system first reconstructs 3D facial expressions and 3D mouth shapes from color and depth input with a multi-linear model. Concurrently a speaker-independent DNN acoustic model is applied to extract phoneme state posterior probabilities (PSPP) from the audio frames. After that, a lip motion regressor refines the 3D mouth shape based on both PSPP and expression weights of the 3D mouth shapes, as well as their confidences. Finally, the refined 3D mouth shape is combined with other parts of the 3D face to generate the final result. The whole process is fully automatic and executed in real time. The key component of our system is a data-driven regresor for modeling the correlation between speech data and mouth shapes. Based on a precaptured database of accurate 3D mouth shapes and associated speech audio from one speaker, the regressor jointly uses the input speech and visual features to refine the mouth shape of a new actor. We also present an improved DNN acoustic model. It not only preserves accuracy but also achieves real-time performance. Our method efficiently fuses visual and acoustic information for 3D facial performance capture. It generates more accurate 3D mouth motions than other approaches that are based on audio or video input only. It also supports video or audio only input for real-time facial animation. We evaluate the performance of our system with speech and facial expressions captured from different actors. Results demonstrate the efficiency and robustness of our method.

3D Facial Animation from Chinese Text.

Text-driven Visual Prosody Generation for Embodied Conversational Agents

Breathing Life into Faces: Speech-driven 3D Facial Animation with Natural Head Pose and Detailed Shape

A Speech-Driven 3-D Lip Synthesis with Realistic Dynamics in Mandarin Chinese

3D Realistic Talking Face Co-Driven by Text and Speech

Quick Customization of Particular Human Face and Muscle-Driven Expression Animation

Facial Expression Animation Based on Physical Model

Animating a Chinese interactive virtual character

A Speech-Driven 3-D Tongue Model with Realistic Movement in Mandarin Chinese.

Facial Animation System for Embedded Application

Vision Based Speech Animation Transferring with Underlying Anatomical Structure

Joint Audio-Text Model for Expressive Speech-Driven 3D Facial Animation

Audio-Driven 3D Facial Animation from In-the-Wild Videos

Real-time Synthesis of Chinese Visual Speech and Facial Expressions Using MPEG-4 FAP Features in a Three-Dimensional Avatar

Speech Driven Facial Animation Using Chinese Mandarin Pronunciation Rules

Speech-Driven 3D Face Animation with Composite and Regional Facial Movements

Individual 3D Face Synthesis Based on Orthogonal Photos and Speech-Driven Facial Animation

Real-time Synthesis of Chinese Visua using MPEG-4 FAP Features in a

Personalized Multi-View Face Animation with Lifelike Textures

Video-audio Driven Real-Time Facial Animation.

Pose-Controllable 3D Facial Animation Synthesis using Hierarchical Audio-Vertex Attention