Abstract:We present a novel neural encoder system for acoustic-to-articulatory inversion. We leverage the Pink Trombone voice synthesizer that reveals articulatory parameters (e.g tongue position and vocal cord configuration). Our system is designed to identify the articulatory features responsible for producing specific acoustic characteristics contained in a neural latent representation. To generate the necessary latent embeddings, we employed two main methodologies. The first was a self-supervised variational autoencoder trained from scratch to reconstruct the input signal at the decoder stage. We conditioned its bottleneck layer with a subnetwork called the "projector," which decodes the voice synthesizer's parameters. The second methodology utilized two pretrained models: EnCodec and Wav2Vec. They eliminate the need to train the encoding process from scratch, allowing us to focus on training the projector network. This approach aimed to explore the potential of these existing models in the context of acoustic-to-articulatory inversion. By reusing the pretrained models, we significantly simplified the data processing pipeline, increasing efficiency and reducing computational overhead. The primary goal of our project was to demonstrate that these neural architectures can effectively encapsulate both acoustic and articulatory features. This prediction-based approach is much faster than traditional methods focused on acoustic feature-based parameter optimization. We validated our models by predicting six different parameters and evaluating them with objective and ViSQOL subjective-equivalent metric using both synthesizer- and human-generated sounds. The results show that the predicted parameters can generate human-like vowel sounds when input into the synthesizer. We provide the dataset, code, and detailed findings to support future research in this field.

Spectrogram Inpainting for Interactive Generation of Instrument Sounds

SING: Symbol-to-Instrument Neural Generator

VampNet: Music Generation via Masked Acoustic Token Modeling

Vision-Infused Deep Audio Inpainting

Images that Sound: Composing Images and Sounds on a Single Canvas

Pictures Of MIDI: Controlled Music Generation via Graphical Prompts for Image-Based Diffusion Inpainting

Decoding Vocal Articulations from Acoustic Latent Representations

VQCPC-GAN: Variable-Length Adversarial Audio Synthesis Using Vector-Quantized Contrastive Predictive Coding

Generative timbre spaces: regularizing variational auto-encoders with perceptual metrics

SynthScribe: Deep Multimodal Tools for Synthesizer Sound Retrieval and Exploration

Neural Audio Synthesis of Musical Notes with WaveNet Autoencoders

Audio inpainting with generative adversarial network

Masked Generative Video-to-Audio Transformers with Enhanced Synchronicity

Speech inpainting: Context-based speech synthesis guided by video

Generating Sample-Based Musical Instruments Using Neural Audio Codec Language Models

Use of speaker recognition approaches for learning and evaluating embedding representations of musical instrument sounds

Creative Text-to-Audio Generation via Synthesizer Programming

Controllable Music Inpainting with Mixed-Level and Disentangled Representation

Interpretable Timbre Synthesis using Variational Autoencoders Regularized on Timbre Descriptors

NONOTO: A Model-agnostic Web Interface for Interactive Music Composition by Inpainting