Abstract:We present a novel neural encoder system for acoustic-to-articulatory inversion. We leverage the Pink Trombone voice synthesizer that reveals articulatory parameters (e.g tongue position and vocal cord configuration). Our system is designed to identify the articulatory features responsible for producing specific acoustic characteristics contained in a neural latent representation. To generate the necessary latent embeddings, we employed two main methodologies. The first was a self-supervised variational autoencoder trained from scratch to reconstruct the input signal at the decoder stage. We conditioned its bottleneck layer with a subnetwork called the "projector," which decodes the voice synthesizer's parameters. The second methodology utilized two pretrained models: EnCodec and Wav2Vec. They eliminate the need to train the encoding process from scratch, allowing us to focus on training the projector network. This approach aimed to explore the potential of these existing models in the context of acoustic-to-articulatory inversion. By reusing the pretrained models, we significantly simplified the data processing pipeline, increasing efficiency and reducing computational overhead. The primary goal of our project was to demonstrate that these neural architectures can effectively encapsulate both acoustic and articulatory features. This prediction-based approach is much faster than traditional methods focused on acoustic feature-based parameter optimization. We validated our models by predicting six different parameters and evaluating them with objective and ViSQOL subjective-equivalent metric using both synthesizer- and human-generated sounds. The results show that the predicted parameters can generate human-like vowel sounds when input into the synthesizer. We provide the dataset, code, and detailed findings to support future research in this field.

Speaker-Independent Acoustic-to-Articulatory Speech Inversion

Speaker-Independent Acoustic-to-Articulatory Inversion through Multi-Channel Attention Discriminator

Unsupervised Acoustic-to-Articulatory Inversion with Variable Vocal Tract Anatomy

Two-Stream Joint-Training for Speaker Independent Acoustic-to-Articulatory Inversion

Acoustic-to-articulatory inversion for dysarthric speech: Are pre-trained self-supervised representations favorable?

Speaker- and Text-Independent Estimation of Articulatory Movements and Phoneme Alignments from Speech

On the Evaluation of Inversion Mapping Performance in the Acoustic Domain

A deep recurrent approach for acoustic-to-articulatory inversion

Articulatory-WaveNet: Autoregressive Model For Acoustic-to-Articulatory Inversion

Decoding Vocal Articulations from Acoustic Latent Representations

DNN-based Acoustic-to-Articulatory Inversion using Ultrasound Tongue Imaging

Estimate Articulatory Mri Series From Acoustic Signal Using Deep Architecture

Research on the Distal Supervised Learning Model of Speech Inversion.

Accent Conversion with Articulatory Representations

Speaker-independent speech inversion for recovery of velopharyngeal port constriction degreea)

A generalized smoothness criterion for acoustic-to-articulatory inversion

Combined Articulatory and Auditory Processing for Improved Speech Recognition

The Secret Source : Incorporating Source Features to Improve Acoustic-to-Articulatory Speech Inversion

Speaker Adaptation on Articulation and Acoustics for Articulation-to-Speech Synthesis

Complete reconstruction of the tongue contour through acoustic to articulatory inversion using real-time MRI data

Improving Speech Inversion Through Self-Supervised Embeddings and Enhanced Tract Variables