Abstract:This paper presents a method to convert articulatory movements into speech waveforms using a data-driven approach. In this method, recorded electromagnetic midsagittal articulography (EMA) measurements are converted into both spectral features (i.e., Mel-cepstra) and excitation features (i.e., power, voiced/unvoiced flag, and F0) from which speech waveforms are then reconstructed. By considering the nonlinear and dynamic dependency relationships between articulatory movements and acoustic signals, this study adopts bidirectional long short-term memory (BLSTM) based recurrent neural networks (RNN) for the articulatory-to-acoustic conversion. Due to the limitations of current data acquisition technology, recorded articulatory movements are inevitably insufficient to completely describe the articulatory configuration during pronunciation. Therefore, this paper proposes to further augment the model input by concatenating EMA vectors with two other representations. First, the posterior probabilities derived from a phoneme classifier are concatenated with EMA features to provide a linguistic description of each frame for acoustic feature prediction. The classifier is trained to determine the phoneme label of each frame based on the observed EMA features. Second, a cascaded prediction strategy is designed to utilize the predicted spectral features as auxiliary input to boost the prediction accuracy of the excitation features. The results of experiment show that BLSTM-RNNs can achieve a better objective and subjective performance than deep neural networks (DNN) and Gaussian mixture models (GMM) in articulatory-to-acoustic conversion. In addition, our results show that the proposed methods for integrating linguistic representation and utilizing a cascaded prediction strategy can further improve the accuracy of acoustic feature prediction.

Utilizing auxiliary data in phoneme recognition based on Articulatory Feature

Accent Recognition with Hybrid Phonetic Features

AudioVSR: Enhancing Video Speech Recognition with Audio Data

Exploiting Articulatory Features for Pitch Accent Detection.

Detection-based accented speech recognition using articulatory features.

Integrating Articulatory Features into Acoustic-Phonemic Model for Mispronunciation Detection and Diagnosis in L2 English Speech.

Improve low-resource non-native mispronunciation detection with native speech by articulatory-based tandem feature

Research on the algorithm of consonant feature extraction for speech training aid

Exploiting Cross-domain And Cross-Lingual Ultrasound Tongue Imaging Features For Elderly And Dysarthric Speech Recognition

Incorporating L2 Phonemes Using Articulatory Features for Robust Speech Recognition

Articulatory-to-acoustic Conversion Using BLSTM-RNNs with Augmented Input Representation.

Integrating Articulatory Features into HMM-Based Parametric Speech Synthesis

Acoustic Modeling Based On Chinese Phonetics Knowledge

Articulatory Feature Based Multilingual MLPs for Low-Resource Speech Recognition.

Strategies for using MLP based features with limited target-language training data.

Language-Independent Approach for Automatic Computation of Vowel Articulation Features in Dysarthric Speech Assessment

An Approach to Mispronunciation Detection and Diagnosis with Acoustic, Phonetic and Linguistic (APL) Embeddings

Accent Conversion with Articulatory Representations

Auxiliary Features from Laser-Doppler Vibrometer Sensor for Deep Neural Network Based Robust Speech Recognition

Feature-Space Transform Tying in Unified Acoustic-Articulatory Modelling for Articulatory Control of HMM-Based Speech Synthesis.

Articulation constrained learning with application to speech emotion recognition