Abstract:Speech enhancement and speech separation are two related tasks, whose purpose is to extract either one or more target speech signals, respectively, from a mixture of sounds generated by several sources. Traditionally, these tasks have been tackled using signal processing and machine learning techniques applied to the available acoustic signals. Since the visual aspect of speech is essentially unaffected by the acoustic environment, visual information from the target speakers, such as lip movements and facial expressions, has also been used for speech enhancement and speech separation systems. In order to efficiently fuse acoustic and visual information, researchers have exploited the flexibility of data-driven approaches, specifically deep learning, achieving strong performance. The ceaseless proposal of a large number of techniques to extract features and fuse multimodal information has highlighted the need for an overview that comprehensively describes and discusses audio-visual speech enhancement and separation based on deep learning. In this paper, we provide a systematic survey of this research topic, focusing on the main elements that characterise the systems in the literature: acoustic features; visual features; deep learning methods; fusion techniques; training targets and objective functions. In addition, we review deep-learning-based methods for speech reconstruction from silent videos and audio-visual sound source separation for non-speech signals, since these methods can be more or less directly applied to audio-visual speech enhancement and separation. Finally, we survey commonly employed audio-visual speech datasets, given their central role in the development of data-driven approaches, and evaluation methods, because they are generally used to compare different systems and determine their performance.

DyViSE: Dynamic Vision-Guided Speaker Embedding for Audio-Visual Speaker Diarization

Late Audio-Visual Fusion for In-The-Wild Speaker Diarization

Uncertainty-Guided End-to-End Audio-Visual Speaker Diarization for Far-Field Recordings

Integrating Audio, Visual, and Semantic Information for Enhanced Multimodal Speaker Diarization

Audio-Visual Speaker Diarization Based on Spatiotemporal Bayesian Fusion

Speaker Embeddings With Weakly Supervised Voice Activity Detection For Efficient Speaker Diarization

Rethinking the visual cues in audio-visual speaker extraction

Multi-View Speaker Embedding Learning for Enhanced Stability and Discriminability.

AVA-AVD: Audio-Visual Speaker Diarization in the Wild

A lightweight approach to real-time speaker diarization: from audio toward audio-visual data streams

Audio–Visual Deep Clustering for Speech Separation

DiffV2S: Diffusion-based Video-to-Speech Synthesis with Vision-guided Speaker Embedding

Joint Training or Not: An Exploration of Pre-trained Speech Models in Audio-Visual Speaker Diarization

Quality-Aware End-to-End Audio-Visual Neural Speaker Diarization

Audio-visual End-to-end Multi-channel Speech Separation, Dereverberation and Recognition

An Overview of Deep-Learning-Based Audio-Visual Speech Enhancement and Separation

Multi-Input Multi-Output Target-Speaker Voice Activity Detection For Unified, Flexible, and Robust Audio-Visual Speaker Diarization

Cooperative Dual Attention for Audio-Visual Speech Enhancement with Facial Cues

Integration of audio-visual information for multi-speaker multimedia speaker recognition

Leveraging Speaker Embeddings in End-to-End Neural Diarization for Two-Speaker Scenarios

Speaker disentanglement in video-to-speech conversion