Abstract:Understanding the relationship between tongue motion patterns during speech and their resulting speech acoustic outcomes -- i.e., articulatory-acoustic relation -- is of great importance in assessing speech quality and developing innovative treatment and rehabilitative strategies. This is especially important when evaluating and detecting abnormal articulatory features in patients with speech-related disorders. In this work, we aim to develop a framework for detecting speech motion anomalies in conjunction with their corresponding speech acoustics. This is achieved through the use of a deep cross-modal translator trained on data from healthy individuals only, which bridges the gap between 4D motion fields obtained from tagged MRI and 2D spectrograms derived from speech acoustic data. The trained translator is used as an anomaly detector, by measuring the spectrogram reconstruction quality on healthy individuals or patients. In particular, the cross-modal translator is likely to yield limited generalization capabilities on patient data, which includes unseen out-of-distribution patterns and demonstrates subpar performance, when compared with healthy individuals.~A one-class SVM is then used to distinguish the spectrograms of healthy individuals from those of patients. To validate our framework, we collected a total of 39 paired tagged MRI and speech waveforms, consisting of data from 36 healthy individuals and 3 tongue cancer patients. We used both 3D convolutional and transformer-based deep translation models, training them on the healthy training set and then applying them to both the healthy and patient testing sets. Our framework demonstrates a capability to detect abnormal patient data, thereby illustrating its potential in enhancing the understanding of the articulatory-acoustic relation for both healthy individuals and patients.

Tongue shape conversion with non-parallel training data

Inverse Biomechanical Modeling of the Tongue via Machine Learning and Synthetic Training Data

A Multilinear Tongue Model Derived from Speech Related MRI Data of the Human Vocal Tract

Enhancing Speech Articulation Analysis using a Geometric Transformation of the X-ray Microbeam Dataset

Transfer the linguistic representations from TTS to accent conversion with non-parallel data

Articulatory-to-acoustic Conversion Using BLSTM-RNNs with Augmented Input Representation.

TTS-Guided Training for Accent Conversion Without Parallel Data

Supervisory Data Alignment for Text-Independent Voice Conversion

A Speech-Driven 3-D Tongue Model with Realistic Movement in Mandarin Chinese.

Complete reconstruction of the tongue contour through acoustic to articulatory inversion using real-time MRI data

Tongue Shape Variation Model for Simulating Mandarin Chinese Articulation.

Decoding the dancing of the tongue: A model-based learning approach to phonetic targets in coarticulationa)

Tongue Postures and Tongue Centers: A Study of Acoustic-Articulatory Correspondences Across Different Head Angles

Extraction of Tongue Contour in Real-Time Magnetic Resonance Imaging Sequences

Construction of An Unbiased Spatio-Temporal Atlas of the Tongue During Speech

Speech motion anomaly detection via cross-modal translation of 4D motion fields from tagged MRI

Tongue Image Alignment via Conformal Mapping for Disease Detection

Non-parallel training for voice conversion based on FT-GMM

Accent Conversion with Articulatory Representations

A statistical shape space model of the palate surface trained on 3D MRI scans of the vocal tract

A mass-spring tongue model with efficient collision detection and response during speech