Abstract:With the development of audio playback devices and fast data transmission, the demand for high sound quality is rising for both entertainment and communications. In this quest for better sound quality, challenges emerge from distortions and interferences originating at the recording side or caused by an imperfect transmission pipeline. To address this problem, audio restoration methods aim to recover clean sound signals from the corrupted input data. We present here audio restoration algorithms based on diffusion models, with a focus on speech enhancement and music restoration tasks. Traditional approaches, often grounded in handcrafted rules and statistical heuristics, have shaped our understanding of audio signals. In the past decades, there has been a notable shift towards data-driven methods that exploit the modeling capabilities of DNNs. Deep generative models, and among them diffusion models, have emerged as powerful techniques for learning complex data distributions. However, relying solely on DNN-based learning approaches carries the risk of reducing interpretability, particularly when employing end-to-end models. Nonetheless, data-driven approaches allow more flexibility in comparison to statistical model-based frameworks, whose performance depends on distributional and statistical assumptions that can be difficult to guarantee. Here, we aim to show that diffusion models can combine the best of both worlds and offer the opportunity to design audio restoration algorithms with a good degree of interpretability and a remarkable performance in terms of sound quality. We explain the diffusion formalism and its application to the conditional generation of clean audio signals. We believe that diffusion models open an exciting field of research with the potential to spawn new audio restoration algorithms that are natural-sounding and remain robust in difficult acoustic situations.

Speech Signal Improvement Using Causal Generative Diffusion Models

Speech Enhancement and Dereverberation with Diffusion-based Generative Models

Investigating the Design Space of Diffusion Models for Speech Enhancement

Single and Few-step Diffusion for Generative Speech Enhancement

ICASSP 2024 Speech Signal Improvement Challenge

A Study on Speech Enhancement Based on Diffusion Probabilistic Model

Two-stage Neural Network for ICASSP 2023 Speech Signal Improvement Challenge

Diffusion Models for Audio Restoration

SRTNET: Time Domain Speech Enhancement Via Stochastic Refinement

Causal Signal-Based DCCRN with Overlapped-Frame Prediction for Online Speech Enhancement

Diffusion-based Speech Enhancement with Schrödinger Bridge and Symmetric Noise Schedule

GALD-SE: Guided Anisotropic Lightweight Diffusion for Efficient Speech Enhancement

StoRM: A Diffusion-based Stochastic Regeneration Model for Speech Enhancement and Dereverberation

Unsupervised speech enhancement with diffusion-based generative models

Inference and Denoise: Causal Inference-based Neural Speech Enhancement

Audio-Visual Speech Enhancement with Score-Based Generative Models

FastDiff 2: Revisiting and Incorporating GANs and Diffusion Models in High-Fidelity Speech Synthesis

Pre-training Feature Guided Diffusion Model for Speech Enhancement

Diffusion-based speech enhancement with a weighted generative-supervised learning loss

Conditional Generative Adversarial Networks for Speech Enhancement and Noise-Robust Speaker Verification