Abstract:In environments characterised by noise or the absence of audio signals, visual cues, notably facial and lip movements, serve as valuable substitutes for missing or corrupted speech signals. In these scenarios, speech reconstruction can potentially generate speech from visual data. Recent advancements in this domain have predominantly relied on end-to-end deep learning models, like Convolutional Neural Networks (CNN) or Generative Adversarial Networks (GAN). However, these models are encumbered by their intricate and opaque architectures, coupled with their lack of speaker independence. Consequently, achieving multi-speaker speech reconstruction without supplementary information is challenging. This research introduces an innovative Gabor-based speech reconstruction system tailored for lightweight and efficient multi-speaker speech restoration. Using our Gabor feature extraction technique, we propose two novel models: GaborCNN2Speech and GaborFea2Speech. These models employ a rapid Gabor feature extraction method to derive lowdimensional mouth region features, encompassing filtered Gabor mouth images and low-dimensional Gabor features as visual inputs. An encoded spectrogram serves as the audio target, and a Long Short-Term Memory (LSTM)-based model is harnessed to generate coherent speech output. Through comprehensive experiments conducted on the GRID corpus, our proposed Gabor-based models have showcased superior performance in sentence and vocabulary reconstruction when compared to traditional end-to-end CNN models. These models stand out for their lightweight design and rapid processing capabilities. Notably, the GaborFea2Speech model presented in this study achieves robust multi-speaker speech reconstruction without necessitating supplementary information, thereby marking a significant milestone in the field of speech reconstruction.

Speech reconstruction using a generalized HSMM (GHSMM)

A GMM/HMM model for reconstruction of missing speech spectral components for continuous speech recognition

A Novel HTS System Using both Continuous HMMs and Discrete HMMs

Why has (reasonably accurate) Automatic Speech Recognition been so hard to achieve?

Speech Synthesis Based on Gaussian Conditional Random Fields

Automatic Speech Recognition : A Study and Performance Evaluation on Neural Networks and Hidden Markov Models

Statistical Parametric Speech Synthesis Using a Hidden Trajectory Model

A Novel Hmm-Based Tts System Using Both Continuous Hmms And Discrete Hmms

A Deep Generative Architecture for Postfiltering in Statistical Parametric Speech Synthesis

Exploiting Audio-Visual Features with Pretrained AV-HuBERT for Multi-Modal Dysarthric Speech Reconstruction

Lip2Speech: Lightweight Multi-Speaker Speech Reconstruction with Gabor Features

Hidden Markov Acoustic Modeling with Bootstrap and Restructuring for Low-Resourced Languages

Learning Hidden Markov Models When the Locations of Missing Observations are Unknown

DNN-based Stochastic Postfilter for HMM-based Speech Synthesis

Duration-Distribution-Based HMM for Speech Recognition

Competitive and Resource Efficient Factored Hybrid HMM Systems are Simpler Than You Think

A hidden trajectory model with bi-directional target filtering: cascaded vs. integrated implementation for phonetic recognition

Improving Factored Hybrid HMM Acoustic Modeling without State Tying

Histogram of States Based Assistive System for Speech Impairment Due to Neurological Disorders

High-Resolution Speech Restoration with Latent Diffusion Model

Modeling Spectral Envelopes Using Deep Conditional Restricted Boltzmann Machines for Statistical Parametric Speech Synthesis.