Abstract:We present a novel neural encoder system for acoustic-to-articulatory inversion. We leverage the Pink Trombone voice synthesizer that reveals articulatory parameters (e.g tongue position and vocal cord configuration). Our system is designed to identify the articulatory features responsible for producing specific acoustic characteristics contained in a neural latent representation. To generate the necessary latent embeddings, we employed two main methodologies. The first was a self-supervised variational autoencoder trained from scratch to reconstruct the input signal at the decoder stage. We conditioned its bottleneck layer with a subnetwork called the "projector," which decodes the voice synthesizer's parameters. The second methodology utilized two pretrained models: EnCodec and Wav2Vec. They eliminate the need to train the encoding process from scratch, allowing us to focus on training the projector network. This approach aimed to explore the potential of these existing models in the context of acoustic-to-articulatory inversion. By reusing the pretrained models, we significantly simplified the data processing pipeline, increasing efficiency and reducing computational overhead. The primary goal of our project was to demonstrate that these neural architectures can effectively encapsulate both acoustic and articulatory features. This prediction-based approach is much faster than traditional methods focused on acoustic feature-based parameter optimization. We validated our models by predicting six different parameters and evaluating them with objective and ViSQOL subjective-equivalent metric using both synthesizer- and human-generated sounds. The results show that the predicted parameters can generate human-like vowel sounds when input into the synthesizer. We provide the dataset, code, and detailed findings to support future research in this field.

Unsupervised Inference of Physiologically Meaningful Articulatory Trajectories with VocalTractLab

Unsupervised Acoustic-to-Articulatory Inversion with Variable Vocal Tract Anatomy

Embodied Self-supervised Learning by Coordinated Sampling and Training

Deep Speech Synthesis from MRI-Based Articulatory Representations

ArtSpeech: Adaptive Text-to-Speech Synthesis with Articulatory Representations

Articulatory Copy Synthesis Based on the Speech Synthesizer VocalTractLab and Convolutional Recurrent Neural Networks

Speaker-Independent Acoustic-to-Articulatory Speech Inversion

Decoding Vocal Articulations from Acoustic Latent Representations

An Adaptive Neural Control Scheme for Articulatory Synthesis of CV Sequences

VISUAL SPEECH SYNTHESIS BASED ON ARTICULATORY TRAJECTORY

Self-Supervised Solution to the Control Problem of Articulatory Synthesis

A deep recurrent approach for acoustic-to-articulatory inversion

Multi-modal Recording and Modeling of Vocal Tract Movements

Vocal Tract Area Estimation by Gradient Descent

Articulatory Phonetics Informed Controllable Expressive Speech Synthesis

A Realistic 3d Articulatory Animation System for Emotional Visual Pronunciation

Speech synthesis of VCV sequence using a physiological articulatory model

The Use of Articulatory Movement Data in Speech Synthesis Applications: an Overview — Application of Articulatory Movements Using Machine Learning Algorithms —

Artificial Vocal Learning Guided by Speech Recognition: What It May Tell Us about How Children Learn to Speak

Reconstructing Speech from Real-Time Articulatory MRI Using Neural Vocoders

Simulating Articulatory Trajectories with Phonological Feature Interpolation