Abstract:In environments characterised by noise or the absence of audio signals, visual cues, notably facial and lip movements, serve as valuable substitutes for missing or corrupted speech signals. In these scenarios, speech reconstruction can potentially generate speech from visual data. Recent advancements in this domain have predominantly relied on end-to-end deep learning models, like Convolutional Neural Networks (CNN) or Generative Adversarial Networks (GAN). However, these models are encumbered by their intricate and opaque architectures, coupled with their lack of speaker independence. Consequently, achieving multi-speaker speech reconstruction without supplementary information is challenging. This research introduces an innovative Gabor-based speech reconstruction system tailored for lightweight and efficient multi-speaker speech restoration. Using our Gabor feature extraction technique, we propose two novel models: GaborCNN2Speech and GaborFea2Speech. These models employ a rapid Gabor feature extraction method to derive lowdimensional mouth region features, encompassing filtered Gabor mouth images and low-dimensional Gabor features as visual inputs. An encoded spectrogram serves as the audio target, and a Long Short-Term Memory (LSTM)-based model is harnessed to generate coherent speech output. Through comprehensive experiments conducted on the GRID corpus, our proposed Gabor-based models have showcased superior performance in sentence and vocabulary reconstruction when compared to traditional end-to-end CNN models. These models stand out for their lightweight design and rapid processing capabilities. Notably, the GaborFea2Speech model presented in this study achieves robust multi-speaker speech reconstruction without necessitating supplementary information, thereby marking a significant milestone in the field of speech reconstruction.

Vid2speech: Speech Reconstruction from Silent Video

Improved Speech Reconstruction from Silent Video

LipVoicer: Generating Speech from Silent Videos Guided by Lip Reading

Visual Speech Enhancement

Seeing Through Noise: Visually Driven Speaker Separation and Enhancement

Speech Reconstruction With Reminiscent Sound Via Visual Voice Memory

SVTS: Scalable Video-to-Speech Synthesis

Lip2AudSpec: Speech reconstruction from silent lip movements video

Speech-Driven Facial Animations Improve Speech-in-Noise Comprehension of Humans

Voice Presentation Attack Detection Using Convolutional Neural Networks

Lip2Speech: Lightweight Multi-Speaker Speech Reconstruction with Gabor Features

Towards Accurate Lip-to-Speech Synthesis in-the-Wild

An arabic visual speech recognition framework with CNN and vision transformers for lipreading

Lip-to-Speech Synthesis for Arbitrary Speakers in the Wild

Synthesizing Audio from Silent Video using Sequence to Sequence Modeling

Robust One Shot Audio to Video Generation

Lipreading Architecture Based on Multiple Convolutional Neural Networks for Sentence-Level Visual Speech Recognition

Let There Be Sound: Reconstructing High Quality Speech from Silent Videos

LA-VocE: Low-SNR Audio-visual Speech Enhancement using Neural Vocoders

Audio-Visual Speech Codecs: Rethinking Audio-Visual Speech Enhancement by Re-Synthesis

Toward end-to-end interpretable convolutional neural networks for waveform signals