Abstract:The goal of speech enhancement (SE) is to eliminate the background interference from the noisy speech signal. Generative models such as diffusion models (DM) have been applied to the task of SE because of better generalization in unseen noisy scenes. Technical routes for the DM-based SE methods can be summarized into three types: task-adapted diffusion process formulation, generator-plus-conditioner (GPC) structures and the multi-stage frameworks. We focus on the first two approaches, which are constructed under the GPC architecture and use the task-adapted diffusion process to better deal with the real noise. However, the performance of these SE models is limited by the following issues: (a) Non-Gaussian noise estimation in the task-adapted diffusion process. (b) Conditional domain bias caused by the weak conditioner design in the GPC structure. (c) Large amount of residual noise caused by unreasonable interpolation operations during inference. To solve the above problems, we propose a noise-aware diffusion-based SE model (NADiffuSE) to boost the SE performance, where the noise representation is extracted from the noisy speech signal and introduced as a global conditional information for estimating the non-Gaussian components. Furthermore, the anchor-based inference algorithm is employed to achieve a compromise between the speech distortion and noise residual. In order to mitigate the performance degradation caused by the conditional domain bias in the GPC framework, we investigate three model variants, all of which can be viewed as multi-stage SE based on the preprocessing networks for Mel spectrograms. Experimental results show that NADiffuSE outperforms other DM-based SE models under the GPC infrastructure. Audio samples are available at: <a class="link-external link-https" href="https://square-of-w.github.io/NADiffuSE-demo/" rel="external noopener nofollow">this https URL</a>.

DiffWave: A Versatile Diffusion Model for Audio Synthesis

DiffAR: Denoising Diffusion Autoregressive Model for Raw Speech Waveform Generation

FastDiff: A Fast Conditional Diffusion Model for High-Quality Speech Synthesis

Diffsound: Discrete Diffusion Model for Text-to-Sound Generation

From Discrete Tokens to High-Fidelity Audio Using Multi-Band Diffusion

FastDiff 2: Revisiting and Incorporating GANs and Diffusion Models in High-Fidelity Speech Synthesis

UnDiff: Unsupervised Voice Restoration with Unconditional Diffusion Model

SpecDiff-GAN: A Spectrally-Shaped Noise Diffusion GAN for Speech and Music Synthesis

GLA-Grad: A Griffin-Lim Extended Waveform Generation Diffusion Model

EDMSound: Spectrogram Based Diffusion Models for Efficient and High-Quality Audio Synthesis

WaveFlow: A Compact Flow-based Model for Raw Audio

Boosting Fast and High-Quality Speech Synthesis with Linear Diffusion

WaveTransfer: A Flexible End-to-end Multi-instrument Timbre Transfer with Diffusion

Mtdiffusion: Multi-Task Diffusion Model With Dual-Unet for Foley Sound Generation.

A Study on Speech Enhancement Based on Diffusion Probabilistic Model

Speaking in Wavelet Domain: A Simple and Efficient Approach to Speed up Speech Diffusion Model

Progressive distillation diffusion for raw music generation

DiffVoice: Text-to-Speech with Latent Diffusion

NADiffuSE: Noise-aware Diffusion-based Model for Speech Enhancement

MM-Diffusion: Learning Multi-Modal Diffusion Models for Joint Audio and Video Generation

RFWave: Multi-band Rectified Flow for Audio Waveform Reconstruction