Abstract:This paper presents an investigation into predicting the movement of a speaker's mouth from text input using hidden Markov models (HMM). A corpus of human articulatory movements, recorded by electromagnetic articulography (EMA), is used to train HMMs. To predict articulatory movements for input text, a suitable model sequence is selected and a maximum-likelihood parameter generation (MLPG) algorithm is used to generate output articulatory trajectories. Unified acoustic-articulatory HMMs are introduced to integrate acoustic features when an acoustic signal is also provided with the input text. Several aspects of this method are analyzed in this paper, including the effectiveness of context-dependent modeling, the role of supplementary acoustic input, and the appropriateness of certain model structures for the unified acoustic-articulatory models. When text is the sole input, we find that fully context-dependent models significantly outperform monophone and quinphone models, achieving an average root mean square (RMS) error of 1.945mm and an average correlation coefficient of 0.600. When both text and acoustic features are given as input to the system, the difference between the performance of quinphone models and fully context-dependent models is no longer significant. The best performance overall is achieved using unified acoustic-articulatory quinphone HMMs with separate clustering of acoustic and articulatory model parameters, a synchronous-state sequence, and a dependent-feature model structure, with an RMS error of 0.900mm and a correlation coefficient of 0.855 on average. Finally, we also apply the same quinphone HMMs to the acoustic-articulatory, or inversion, mapping problem, where only acoustic input is available. An average root mean square (RMS) error of 1.076mm and an average correlation coefficient of 0.812 are achieved. Taken together, our results demonstrate how text and acoustic inputs both contribute to the prediction of articulatory movements in the method used.

Simulating Articulatory Trajectories with Phonological Feature Interpolation

Decoding the dancing of the tongue: A model-based learning approach to phonetic targets in coarticulationa)

Decoding the Dancing of the Tongue: A Model-Based Learning Approach to Phonetic Targets in Coarticulation

Integrating Articulatory Features into HMM-Based Parametric Speech Synthesis

Learning Model-Based F0 Production Through Goal-Directed Babbling

Speaker- and Text-Independent Estimation of Articulatory Movements and Phoneme Alignments from Speech

Unsupervised Inference of Physiologically Meaningful Articulatory Trajectories with VocalTractLab

A Realistic 3d Articulatory Animation System for Emotional Visual Pronunciation

The Use of Articulatory Movement Data in Speech Synthesis Applications: an Overview — Application of Articulatory Movements Using Machine Learning Algorithms —

Feature-Space Transform Tying in Unified Acoustic-Articulatory Modelling for Articulatory Control of HMM-Based Speech Synthesis.

Coding Speech through Vocal Tract Kinematics

Novel Acoustic Modeling with Structured Hidden Dynamics for Speech Coarticulation and Reduction

Implement of Coarticulation in Physiological Articulatory Model

An Analysis of HMM-based Prediction of Articulatory Movements

An Adaptive Neural Control Scheme for Articulatory Synthesis of CV Sequences

A Model-Based Learning Process for Modeling Coarticulation of Human Speech

Observation and Modeling of Lingual Coarticulation in the Planning Stage

Speech synthesis using a physiological articulatory model with feature-based rules

On the Emergence of Phonological Knowledge and on Motor Planning and Motor Programming in a Developmental Model of Speech Production

Evaluating Features and Metrics for High-Quality Simulation of Early Vocal Learning of Vowels

Combined X-ray and facial videos for phoneme-level articulator dynamics