Abstract:Human speech processing is inherently multimodal, where visual cues (lip movements) help to better understand the speech in noise. Lip-reading driven speech enhancement significantly outperforms benchmark audio-only approaches at low signal-to-noise ratios (SNRs). However, at high SNRs or low levels of background noise, visual cues become fairly less effective for speech enhancement. Therefore, a more optimal, context-aware audio-visual (AV) system is required, that contextually utilises both visual and noisy audio features and effectively accounts for different noisy conditions. In this paper, we introduce a novel contextual AV switching component that contextually exploits AV cues with respect to different operating conditions to estimate clean audio, without requiring any SNR estimation. The switching module switches between visual-only (V-only), audio-only (A-only), and both AV cues at low, high and moderate SNR levels, respectively. The contextual AV switching component is developed by integrating a convolutional neural network and long-short-term memory network. For testing, the estimated clean audio features are utilised by the developed novel enhanced visually derived Wiener filter for clean audio power spectrum estimation. The contextual AV speech enhancement method is evaluated under real-world scenarios using benchmark Grid and ChiME3 corpora. For objective testing, perceptual evaluation of speech quality is used to evaluate the quality of the restored speech. For subjective testing, the standard mean-opinion-score method is used. The critical analysis and comparative study demonstrate the outperformance of proposed contextual AV approach, over A-only, V-only, spectral subtraction, and log-minimum mean square error based speech enhancement methods at both low and high SNRs, revealing its capability to tackle spectro-temporal variation in any real-world noisy condition.

Robust Real-time Audio-Visual Speech Enhancement based on DNN and GAN

Audio-Visual Speech Enhancement with Deep Multi-modality Fusion

CochleaNet: A Robust Language-independent Audio-Visual Model for Speech Enhancement

Real-Time Audio-Visual End-to-End Speech Enhancement

Conditional Generative Adversarial Networks for Speech Enhancement and Noise-Robust Speaker Verification

Towards Intelligibility-Oriented Audio-Visual Speech Enhancement

SE-MelGAN -- Speaker Agnostic Rapid Speech Enhancement

VSEGAN: Visual Speech Enhancement Generative Adversarial Network

LA-VocE: Low-SNR Audio-visual Speech Enhancement using Neural Vocoders

LSTMSE-Net: Long Short Term Speech Enhancement Network for Audio-visual Speech Enhancement

AeGAN: Time-Frequency Speech Denoising via Generative Adversarial Networks

Generative Adversarial Networks (GANs) for Audio-Visual Speech Recognition in Artificial Intelligence IoT

Improving Deep Neural Network Based Speech Enhancement in Low SNR Environments

SEGAN: Speech Enhancement Generative Adversarial Network

Time-domain Speech Super-resolution with GAN based Modeling for Telephony Speaker Verification

An Empirical Study of Visual Features for DNN based Audio-Visual Speech Enhancement in Multi-talker Environments

Multi-Metric Optimization using Generative Adversarial Networks for Near-End Speech Intelligibility Enhancement

Context-Aware Audio-Visual Speech Enhancement Based on Neuro-Fuzzy Modeling and User Preference Learning

Audio-Visual Speech Enhancement Using Self-supervised Learning to Improve Speech Intelligibility in Cochlear Implant Simulations

Contextual Audio-Visual Switching For Speech Enhancement in Real-World Environments

SEFGAN: Harvesting the Power of Normalizing Flows and GANs for Efficient High-Quality Speech Enhancement