Abstract:In this paper, a new speech driven 3-D geometric tongue model is constructed. The constructed 3-D tongue shape is controlled with control points on 2-D midsagittal tongue curve, and speech-driven inverse estimation based on the constructed model is evaluated by empirical data. X-Ray 2-D vocal tract motion videos are tagged for the midsagittal tongue motion, and static 3-D vocal tracts of 20 phonemes are collected with MRI for the realistic 3-D tongue shape. MFCC are calculated from the videos as acoustic features, and are then used in a LSTM-RNN to predict the control points movement of the tongue shape. Three geometrically intuitive control points are selected to represent and calculate the midsagittal line of the tongue through linear regression. Cross-sections on the central lines of the tongues, whose height, width and angle are then predicted from the midsagittal line, are reconstructed with geometric curves, and the shape of each cross-section are then placed on the midsagittal line to get the overall predicted moving grid of the 3-D tongue. In this 3-D tongue model, acoustic features and realistic tongue motion are mapped directly to preserve more realistic articulatory details, and the control points are intuitive for non-experts to control the model, and the geometric tongue shapes predicted are comparable with realistic tongue dynamics. Based on the proposed method, the speech-driven prediction is evaluated with the realistic data, which proved this proposed method feasible.

A mass-spring tongue model with efficient collision detection and response during speech

A Speech-Driven 3-D Tongue Model with Realistic Movement in Mandarin Chinese.

The Construction of Mass-Spring Model of Cleft Lips

Acoustic VR in the Mouth: A Real-Time Speech-Driven Visual Tongue System.

Tongue–Computer Interface Prototype Design Based on T-Type Magnet Localization for Smart Environment Control

Speech animation using electromagnetic articulography as motion capture data

Progress in animation of an EMA-controlled tongue model for acoustic-visual speech synthesis

Inverse Biomechanical Modeling of the Tongue via Machine Learning and Synthetic Training Data

A 3D dynamical biomechanical tongue model to study speech motor control

A Multilinear Tongue Model Derived from Speech Related MRI Data of the Human Vocal Tract

Synthesizing 3D Trump: Predicting and Visualizing the Relationship Between Text, Speech, and Articulatory Movements.

A Speech-Driven 3-D Lip Synthesis with Realistic Dynamics in Mandarin Chinese

Facial Expression Animation Based on Physical Model

Tongue Movements and Mouth Expressions Animation Based on Multi-curve Spectrum

3D Facial Animation from Chinese Text.

Speaker-independent Lips and Tongue Visualization of Vowels

Tongue Shape Variation Model for Simulating Mandarin Chinese Articulation.

Contour-based 3d tongue motion visualization using ultrasound image sequences

3D muscle-controlled tongue and mouth model

Real-time speech-driven lip synchronization

An articulatory model of standard Chinese using MRI and X-ray movie