Abstract:In this paper, a new speech driven 3-D geometric tongue model is constructed. The constructed 3-D tongue shape is controlled with control points on 2-D midsagittal tongue curve, and speech-driven inverse estimation based on the constructed model is evaluated by empirical data. X-Ray 2-D vocal tract motion videos are tagged for the midsagittal tongue motion, and static 3-D vocal tracts of 20 phonemes are collected with MRI for the realistic 3-D tongue shape. MFCC are calculated from the videos as acoustic features, and are then used in a LSTM-RNN to predict the control points movement of the tongue shape. Three geometrically intuitive control points are selected to represent and calculate the midsagittal line of the tongue through linear regression. Cross-sections on the central lines of the tongues, whose height, width and angle are then predicted from the midsagittal line, are reconstructed with geometric curves, and the shape of each cross-section are then placed on the midsagittal line to get the overall predicted moving grid of the 3-D tongue. In this 3-D tongue model, acoustic features and realistic tongue motion are mapped directly to preserve more realistic articulatory details, and the control points are intuitive for non-experts to control the model, and the geometric tongue shapes predicted are comparable with realistic tongue dynamics. Based on the proposed method, the speech-driven prediction is evaluated with the realistic data, which proved this proposed method feasible.

Visualization of Mandarin articulation by using a physiological articulatory model

Visualization of mandarin articulation driven by ultrasound data

A novel 3D geometric articulatory model

An Emotional Text-Driven 3D Visual Pronunciation System for Mandarin Chinese

A Novel Method for Constructing 3d Geometric Articulatory Models

An articulatory model of standard Chinese using MRI and X-ray movie

Speech production of vowel sequences using a physiological articulatory model

A Speech-Driven 3-D Tongue Model with Realistic Movement in Mandarin Chinese.

A Study of Mandarin Chinese Using X-Ray and MRI

Three-Dimensional Tongue Modeling Based on PCA for Visualized Pronunciation Feedback

A Realistic 3d Articulatory Animation System for Emotional Visual Pronunciation

Improvement of a Physiological Articulatory Model for Synthesis of Vowel Sequences

Estimation of vocal tract shapes from speech sounds with a physiological articulatory model

Acoustic VR in the Mouth: A Real-Time Speech-Driven Visual Tongue System.

Speech synthesis of VCV sequence using a physiological articulatory model

Improvements of a Physiological Articulatory Model in Construction and Control Strategy

Synthesizing 3D Trump: Predicting and Visualizing the Relationship Between Text, Speech, and Articulatory Movements.

A Physiological Model of the Tongue and Jaw for Simulating Deformation in the Midsagittal and Parasagittal Planes

Tongue Shape Variation Model for Simulating Mandarin Chinese Articulation.

Mandarin Vowel Synthesis Based on 2D and 3D Vocal Tract Model by Finite-Difference Time-Domain Method

VISUAL SPEECH SYNTHESIS BASED ON ARTICULATORY TRAJECTORY