Abstract:In this work, we build upon our previous publication and use diffusion-based generative models for speech enhancement. We present a detailed overview of the diffusion process that is based on a stochastic differential equation and delve into an extensive theoretical examination of its implications. Opposed to usual conditional generation tasks, we do not start the reverse process from pure Gaussian noise but from a mixture of noisy speech and Gaussian noise. This matches our forward process which moves from clean speech to noisy speech by including a drift term. We show that this procedure enables using only 30 diffusion steps to generate high-quality clean speech estimates. By adapting the network architecture, we are able to significantly improve the speech enhancement performance, indicating that the network, rather than the formalism, was the main limitation of our original approach. In an extensive cross-dataset evaluation, we show that the improved method can compete with recent discriminative models and achieves better generalization when evaluating on a different corpus than used for training. We complement the results with an instrumental evaluation using real-world noisy recordings and a listening experiment, in which our proposed method is rated best. Examining different sampler configurations for solving the reverse process allows us to balance the performance and computational speed of the proposed method. Moreover, we show that the proposed method is also suitable for dereverberation and thus not limited to additive background noise removal. Code and audio examples are available online, see <a class="link-external link-https" href="https://github.com/sp-uhh/sgmse" rel="external noopener nofollow">this https URL</a>

Investigating Training Objectives for Generative Speech Enhancement

Schrödinger Bridge for Generative Speech Enhancement

Speech Enhancement and Dereverberation with Diffusion-based Generative Models

Diffusion-based Speech Enhancement with Schrödinger Bridge and Symmetric Noise Schedule

Investigating the Design Space of Diffusion Models for Speech Enhancement

Audio-Visual Speech Enhancement with Score-Based Generative Models

Unifying Robustness and Fidelity: A Comprehensive Study of Pretrained Generative Methods for Speech Enhancement in Adverse Conditions

Single and Few-step Diffusion for Generative Speech Enhancement

On the Behavior of Intrusive and Non-intrusive Speech Enhancement Metrics in Predictive and Generative Settings

Schrodinger Bridges Beat Diffusion Models on Text-to-Speech Synthesis

StoRM: A Diffusion-based Stochastic Regeneration Model for Speech Enhancement and Dereverberation

An Objective Evaluation Framework for Pathological Speech Synthesis

Low-latency Speech Enhancement via Speech Token Generation

Multi-Metric Optimization using Generative Adversarial Networks for Near-End Speech Intelligibility Enhancement

Characterization of Deep Learning-Based Speech-Enhancement Techniques in Online Audio Processing Applications

A Two-Stage Training Framework for Joint Speech Compression and Enhancement

Exploring Speech Enhancement with Generative Adversarial Networks for Robust Speech Recognition

Pre-training Feature Guided Diffusion Model for Speech Enhancement

Towards Generalized Speech Enhancement with Generative Adversarial Networks

Study of GANs for Noisy Speech Simulation from Clean Speech

Revisiting Denoising Diffusion Probabilistic Models for Speech Enhancement: Condition Collapse, Efficiency and Refinement