Abstract:This paper presents an investigation into predicting the movement of a speaker's mouth from text input using hidden Markov models (HMM). A corpus of human articulatory movements, recorded by electromagnetic articulography (EMA), is used to train HMMs. To predict articulatory movements for input text, a suitable model sequence is selected and a maximum-likelihood parameter generation (MLPG) algorithm is used to generate output articulatory trajectories. Unified acoustic-articulatory HMMs are introduced to integrate acoustic features when an acoustic signal is also provided with the input text. Several aspects of this method are analyzed in this paper, including the effectiveness of context-dependent modeling, the role of supplementary acoustic input, and the appropriateness of certain model structures for the unified acoustic-articulatory models. When text is the sole input, we find that fully context-dependent models significantly outperform monophone and quinphone models, achieving an average root mean square (RMS) error of 1.945mm and an average correlation coefficient of 0.600. When both text and acoustic features are given as input to the system, the difference between the performance of quinphone models and fully context-dependent models is no longer significant. The best performance overall is achieved using unified acoustic-articulatory quinphone HMMs with separate clustering of acoustic and articulatory model parameters, a synchronous-state sequence, and a dependent-feature model structure, with an RMS error of 0.900mm and a correlation coefficient of 0.855 on average. Finally, we also apply the same quinphone HMMs to the acoustic-articulatory, or inversion, mapping problem, where only acoustic input is available. An average root mean square (RMS) error of 1.076mm and an average correlation coefficient of 0.812 are achieved. Taken together, our results demonstrate how text and acoustic inputs both contribute to the prediction of articulatory movements in the method used.

A New Combined Model of Statics-Dynamics of Speech.

A combined model of statics-dynamics of speech optimized using maximum mutual information

From Linear Prediction HMM to a New Combined Model for Speech Recognition

Discriminative Combination of Multiple Linear Predictions for Speech Recognition.

Improvement of hidden Markov model (HMM) for speech recognition

Parametric model of introducing inter-frame correlation information into hidden markov model for speech recognition

Discriminative Dynamic Gaussian Mixture Selection with Enhanced Robustness and Performance for Multi-Accent Speech Recognition

An New Method Used in HMM for Modeling Frame Correlation.

A New Model for Speech Recognition : Center-Distance Continuous Probability Model

Duration-Distribution-Based HMM for Speech Recognition

Probabilistic Speaker-Class Based Acoustic Modeling for Large Vocabulary Continuous Speech Recognition

Evaluation of parameter generation using high order dynamic features and long span windows for HMM based speech synthesis

Research on Polynomial-Fitting Speech-Trajectory Model in Chinese Continuous Speech Recognition

An inhomogeneous HMM speech recognition algorithm

Dynamic Speaker Selected Training for Rapid Speaker Adaptation

Partial-tied-mixture Auxiliary Chain Models for Speech Recognition Based on Dynamic Bayesian Networks

An Analysis of HMM-based Prediction of Articulatory Movements

A New Subspace Based Speaker Adaptation Method

Improved HMM Model Using Spatial Correlation

Improvement of Probabilistic Acoustic Tube model for speech decomposition

Compensation for the Homogeneous Assumption of Classical HMM by Probability Penalty