Abstract:Understanding the relationship between tongue motion patterns during speech and their resulting speech acoustic outcomes -- i.e., articulatory-acoustic relation -- is of great importance in assessing speech quality and developing innovative treatment and rehabilitative strategies. This is especially important when evaluating and detecting abnormal articulatory features in patients with speech-related disorders. In this work, we aim to develop a framework for detecting speech motion anomalies in conjunction with their corresponding speech acoustics. This is achieved through the use of a deep cross-modal translator trained on data from healthy individuals only, which bridges the gap between 4D motion fields obtained from tagged MRI and 2D spectrograms derived from speech acoustic data. The trained translator is used as an anomaly detector, by measuring the spectrogram reconstruction quality on healthy individuals or patients. In particular, the cross-modal translator is likely to yield limited generalization capabilities on patient data, which includes unseen out-of-distribution patterns and demonstrates subpar performance, when compared with healthy individuals.~A one-class SVM is then used to distinguish the spectrograms of healthy individuals from those of patients. To validate our framework, we collected a total of 39 paired tagged MRI and speech waveforms, consisting of data from 36 healthy individuals and 3 tongue cancer patients. We used both 3D convolutional and transformer-based deep translation models, training them on the healthy training set and then applying them to both the healthy and patient testing sets. Our framework demonstrates a capability to detect abnormal patient data, thereby illustrating its potential in enhancing the understanding of the articulatory-acoustic relation for both healthy individuals and patients.

MRI2Speech: Speech Synthesis from Articulatory Movements Recorded by Real-time MRI

Deep Speech Synthesis from MRI-Based Articulatory Representations

Reconstructing Speech from Real-Time Articulatory MRI Using Neural Vocoders

Speech2rtMRI: Speech-Guided Diffusion Model for Real-time MRI Video of the Vocal Tract during Speech

Speech neuromuscular decoding based on spectrogram images using conformal predictors with Bi-LSTM.

Silent Speech and Emotion Recognition from Vocal Tract Shape Dynamics in Real-Time MRI

A multispeaker dataset of raw and reconstructed speech production real-time MRI video and 3D volumetric images

Deep Speech Synthesis from Multimodal Articulatory Representations

Speaker dependent articulatory-to-acoustic mapping using real-time MRI of the vocal tract

Tagged-MRI Sequence to Audio Synthesis via Self Residual Attention Guided Heterogeneous Translator

Speech Audio Synthesis from Tagged MRI and Non-Negative Matrix Factorization via Plastic Transformer

Synthesizing audio from tongue motion during speech using tagged MRI via transformer

Enhancing linguistic research through 2-mm isotropic 3D dynamic speech MRI optimized by sparse temporal sampling and low-rank reconstruction

Towards Improving NAM-to-Speech Synthesis Intelligibility using Self-Supervised Speech Models

Speech motion anomaly detection via cross-modal translation of 4D motion fields from tagged MRI

An Error Correction Scheme for Improved Air-Tissue Boundary in Real-Time MRI Video for Speech Production

Using Automatic Speech Recognition to Measure the Intelligibility of Speech Synthesized from Brain Signals

Integrated Magnetic Resonance Imaging Methods for Speech Science and Technology

Real-time Synthesis of Imagined Speech Processes from Minimally Invasive Recordings of Neural Activity

Ethics of Generating Synthetic MRI Vocal Tract Views from the Face

Advancing NAM-to-Speech Conversion with Novel Methods and the MultiNAM Dataset