A Conditional Generative Model for Speech Enhancement

LiZeng-Xi,DaiLi-Rong,SongYan,McloughlinIan

DOI: https://doi.org/10.5555/3288801.3288811

IF: 2.311

2018-01-01

Circuits Systems and Signal Processing

Abstract:Deep learning-based speech enhancement approaches like deep neural networks (DNN) and Long Short-Term Memory (LSTM) have already demonstrated superior results to classical methods. However, these m...

What problem does this paper attempt to address?

Low-latency Speech Enhancement via Speech Token Generation

Huaying Xue,Xiulian Peng,Yan Lu

2024-01-23

Abstract:Existing deep learning based speech enhancement mainly employ a data-driven approach, which leverage large amounts of data with a variety of noise types to achieve noise removal from noisy signal. However, the high dependence on the data limits its generalization on the unseen complex noises in real-life environment. In this paper, we focus on the low-latency scenario and regard speech enhancement as a speech generation problem conditioned on the noisy signal, where we generate clean speech instead of identifying and removing noises. Specifically, we propose a conditional generative framework for speech enhancement, which models clean speech by acoustic codes of a neural speech codec and generates the speech codes conditioned on past noisy frames in an auto-regressive way. Moreover, we propose an explicit-alignment approach to align noisy frames with the generated speech tokens to improve the robustness and scalability to different input lengths. Different from other methods that leverage multiple stages to generate speech codes, we leverage a single-stage speech generation approach based on the TF-Codec neural codec to achieve high speech quality with low latency. Extensive results on both synthetic and real-recorded test set show its superiority over data-driven approaches in terms of noise robustness and temporal speech coherence.

Sound,Multimedia,Audio and Speech Processing
Unifying Robustness and Fidelity: A Comprehensive Study of Pretrained Generative Methods for Speech Enhancement in Adverse Conditions

Heming Wang,Meng Yu,Hao Zhang,Chunlei Zhang,Zhongweiyang Xu,Muqiao Yang,Yixuan Zhang,Dong Yu

2023-09-16

Abstract:Enhancing speech signal quality in adverse acoustic environments is a persistent challenge in speech processing. Existing deep learning based enhancement methods often struggle to effectively remove background noise and reverberation in real-world scenarios, hampering listening experiences. To address these challenges, we propose a novel approach that uses pre-trained generative methods to resynthesize clean, anechoic speech from degraded inputs. This study leverages pre-trained vocoder or codec models to synthesize high-quality speech while enhancing robustness in challenging scenarios. Generative methods effectively handle information loss in speech signals, resulting in regenerated speech that has improved fidelity and reduced artifacts. By harnessing the capabilities of pre-trained models, we achieve faithful reproduction of the original speech in adverse conditions. Experimental evaluations on both simulated datasets and realistic samples demonstrate the effectiveness and robustness of our proposed methods. Especially by leveraging codec, we achieve superior subjective scores for both simulated and realistic recordings. The generated speech exhibits enhanced audio quality, reduced background noise, and reverberation. Our findings highlight the potential of pre-trained generative techniques in speech processing, particularly in scenarios where traditional methods falter. Demos are available at <a class="link-external link-https" href="https://whmrtm.github.io/SoundResynthesis" rel="external noopener nofollow">this https URL</a>.

Audio and Speech Processing,Sound
PAGAN: A Phase-Adapted Generative Adversarial Networks for Speech Enhancement

Peishuo Li,Zihang Jiang,Shouyi Yin,Dandan Song,Peng Ouyang,Leibo Liu,Shaojun Wei

DOI: https://doi.org/10.1109/icassp40776.2020.9054256

2020-01-01

Abstract:Deep neural networks (DNNs) are becoming more and more popular in speech enhancement. Most of DNN-based speech enhancement approaches currently operate on magnitude spectra and ignore the phase mismatch between noisy and clean speech which greatly limits the speech enhancement performance. This paper presents a new approach to solve the phase mismatch problem by training traditional DNN adversarially with a time-domain discriminator. Instead of estimating a more accurate phase, the DNN is trained to be more adapted to noisy phase and able to minimize the influence brought by the phase mismatch. We also propose a new evaluation metric to judge the degree of adaptation to noisy phase. Experimental results show that adding of time-domain discriminator yields a more phase-adapted generator and significantly improves the speech enhancement performance.
DCCRGAN: Deep Complex Convolution Recurrent Generator Adversarial Network for Speech Enhancement

Huixiang Huang,Renjie Wu,Jingbiao Huang,Jucai Lin,Jun Yin

DOI: https://doi.org/10.48550/arXiv.2012.10732

2021-03-07

Abstract:Generative adversarial network (GAN) still exists some problems in dealing with speech enhancement (SE) task. Some GAN-based systems adopt the same structure from Pixel-to-Pixel directly without special optimization. The importance of the generator network has not been fully explored. Other related researches change the generator network but operate in the time-frequency domain, which ignores the phase mismatch problem. In order to solve these problems, a deep complex convolution recurrent GAN (DCCRGAN) structure is proposed in this paper. The complex module builds the correlation between magnitude and phase of the waveform and has been proved to be effective. The proposed structure is trained in an end-to-end way. Different LSTM layers are used in the generator network to sufficiently explore the speech enhancement performance of DCCRGAN. The experimental results confirm that the proposed DCCRGAN outperforms the state-of-the-art GAN-based SE systems.

Audio and Speech Processing,Sound
An Experimental Study on Speech Enhancement Based on Deep Neural Networks

Yong Xu,Jun Du,Li-Rong Dai,Chin-Hui Lee

DOI: https://doi.org/10.1109/lsp.2013.2291240

2014-01-01

IEEE Signal Processing Letters

Abstract:This letter presents a regression-based speech enhancement framework using deep neural networks (DNNs) with a multiple-layer deep architecture. In the DNN learning process, a large training set ensures a powerful modeling capability to estimate the complicated nonlinear mapping from observed noisy speech to desired clean signals. Acoustic context was found to improve the continuity of speech to be separated from the background noises successfully without the annoying musical artifact commonly observed in conventional speech enhancement algorithms. A series of pilot experiments were conducted under multi-condition training with more than 100 hours of simulated speech data, resulting in a good generalization capability even in mismatched testing conditions. When compared with the logarithmic minimum mean square error approach, the proposed DNN-based algorithm tends to achieve significant improvements in terms of various objective quality measures. Furthermore, in a subjective preference evaluation with 10 listeners, 76.35% of the subjects were found to prefer DNN-based enhanced speech to that obtained with other conventional technique.
A regression approach to speech enhancement based on deep neural networks

Yong Xu,Jun Du,Li-Rong Dai,Chin-Hui Lee

DOI: https://doi.org/10.1109/TASLP.2014.2364452

2015-01-01

Abstract:In contrast to the conventional minimum mean square error (MMSE)-based noise reduction techniques, we propose a supervised method to enhance speech by means of finding a mapping function between noisy and clean speech signals based on deep neural networks (DNNs). In order to be able to handle a wide range of additive noises in real-world situations, a large training set that encompasses many possible combinations of speech and noise types, is first designed. A DNN architecture is then employed as a nonlinear regression function to ensure a powerful modeling capability. Several techniques have also been proposed to improve the DNN-based speech enhancement system, including global variance equalization to alleviate the over-smoothing problem of the regression model, and the dropout and noise-aware training strategies to further improve the generalization capability of DNNs to unseen noise conditions. Experimental results demonstrate that the proposed framework can achieve significant improvements in both objective and subjective measures over the conventional MMSE based technique. It is also interesting to observe that the proposed DNN approach can well suppress highly nonstationary noise, which is tough to handle in general. Furthermore, the resulting DNN model, trained with artificial synthesized data, is also effective in dealing with noisy speech data recorded in real-world scenarios without the generation of the annoying musical artifact commonly observed in conventional enhancement methods.

engineering, electrical & electronic,acoustics
Audio-Visual Speech Enhancement with Score-Based Generative Models

Julius Richter,Simone Frintrop,Timo Gerkmann

2023-06-02

Abstract:This paper introduces an audio-visual speech enhancement system that leverages score-based generative models, also known as diffusion models, conditioned on visual information. In particular, we exploit audio-visual embeddings obtained from a self-super\-vised learning model that has been fine-tuned on lipreading. The layer-wise features of its transformer-based encoder are aggregated, time-aligned, and incorporated into the noise conditional score network. Experimental evaluations show that the proposed audio-visual speech enhancement system yields improved speech quality and reduces generative artifacts such as phonetic confusions with respect to the audio-only equivalent. The latter is supported by the word error rate of a downstream automatic speech recognition model, which decreases noticeably, especially at low input signal-to-noise ratios.

Audio and Speech Processing,Machine Learning
GAN-Based Speech Enhancement for Low SNR Using Latent Feature Conditioning

Shrishti Saha Shetu,Emanuël A. P. Habets,Andreas Brendel

2024-10-17

Abstract:Enhancing speech quality under adverse SNR conditions remains a significant challenge for discriminative deep neural network (DNN)-based approaches. In this work, we propose DisCoGAN, which is a time-frequency-domain generative adversarial network (GAN) conditioned by the latent features of a discriminative model pre-trained for speech enhancement in low SNR scenarios. Our proposed method achieves superior performance compared to state-of-the-arts discriminative methods and also surpasses end-to-end (E2E) trained GAN models. We also investigate the impact of various configurations for conditioning the proposed GAN model with the discriminative model and assess their influence on enhancing speech quality

Audio and Speech Processing,Sound,Signal Processing
Unsupervised speech enhancement with deep dynamical generative speech and noise models

Xiaoyu Lin,Simon Leglaive,Laurent Girin,Xavier Alameda-Pineda

2023-06-13

Abstract:This work builds on a previous work on unsupervised speech enhancement using a dynamical variational autoencoder (DVAE) as the clean speech model and non-negative matrix factorization (NMF) as the noise model. We propose to replace the NMF noise model with a deep dynamical generative model (DDGM) depending either on the DVAE latent variables, or on the noisy observations, or on both. This DDGM can be trained in three configurations: noise-agnostic, noise-dependent and noise adaptation after noise-dependent training. Experimental results show that the proposed method achieves competitive performance compared to state-of-the-art unsupervised speech enhancement methods, while the noise-dependent training configuration yields a much more time-efficient inference process.

Audio and Speech Processing,Machine Learning,Sound
Speech enhancement using progressive learning-based convolutional recurrent neural network

Andong Li,Minmin Yuan,Chengshi Zheng,Xiaodong Li

DOI: https://doi.org/10.1016/j.apacoust.2020.107347

IF: 3.614

2020-09-01

Applied Acoustics

Abstract:<p>Recently, progressive learning has shown its capacity to improve speech quality and speech intelligibility when it is combined with deep neural network (DNN) and long short-term memory (LSTM) based monaural speech enhancement algorithms, especially in low signal-to-noise ratio (SNR) conditions. Nevertheless, due to a large number of parameters and high computational complexity, it is hard to implement in current resource-limited micro-controllers and thus, it is essential to significantly reduce both the number of parameters and the computational load for practical applications. For this purpose, we propose a novel progressive learning framework with causal convolutional recurrent neural networks called PL-CRNN, which takes advantage of both convolutional neural networks and recurrent neural networks to drastically reduce the number of parameters and simultaneously improve speech quality and speech intelligibility. Numerous experiments verify the effectiveness of the proposed PL-CRNN model and indicate that it yields consistent better performance than the PL-DNN and PL-LSTM algorithms and also it gets results close even better than the CRNN in terms of objective measurements. Compared with PL-DNN, PL-LSTM, and CRNN, the proposed PL-CRNN algorithm can reduce the number of parameters up to 93%, 97%, and 92%, respectively.</p>

acoustics
Conditional Generative Adversarial Networks for Speech Enhancement and Noise-Robust Speaker Verification

Daniel Michelsanti,Zheng-Hua Tan

DOI: https://doi.org/10.21437/Interspeech.2017-1620

2017-09-07

Abstract:Improving speech system performance in noisy environments remains a challenging task, and speech enhancement (SE) is one of the effective techniques to solve the problem. Motivated by the promising results of generative adversarial networks (GANs) in a variety of image processing tasks, we explore the potential of conditional GANs (cGANs) for SE, and in particular, we make use of the image processing framework proposed by Isola et al. [1] to learn a mapping from the spectrogram of noisy speech to an enhanced counterpart. The SE cGAN consists of two networks, trained in an adversarial manner: a generator that tries to enhance the input noisy spectrogram, and a discriminator that tries to distinguish between enhanced spectrograms provided by the generator and clean ones from the database using the noisy spectrogram as a condition. We evaluate the performance of the cGAN method in terms of perceptual evaluation of speech quality (PESQ), short-time objective intelligibility (STOI), and equal error rate (EER) of speaker verification (an example application). Experimental results show that the cGAN method overall outperforms the classical short-time spectral amplitude minimum mean square error (STSA-MMSE) SE algorithm, and is comparable to a deep neural network-based SE approach (DNN-SE).

Audio and Speech Processing,Machine Learning,Sound,Signal Processing
Mixed T-domain and TF-domain Magnitude and Phase representations for GAN-based speech enhancement

Xin Lin,Yang Zhang,Shiyuan Wang

DOI: https://doi.org/10.1038/s41598-024-68708-w

IF: 4.6

2024-08-02

Scientific Reports

Abstract:Deep learning has made significant advancements in speech enhancement, which plays a crucial role in improving the quality of speech signals in noisy conditions. In this paper, we propose a new approach called M-DGAN, which introduces a time (T)-domain encoder-decoder structure with rich channel representations into the time-frequency (TF)-domain generator framework, resulting in a new generator structure with mixed magnitude and phase representations in the T and TF-domains. The proposed mixed T-domain and TF-domain generator, incorporating the cascaded reworked conformer (CRC) structure, exhibits improved modeling capability and adaptability. Test results on the Voice Bank + DEMAND public dataset show that our method achieves the highest score with and performs well on all the remaining metrics when compared to the current state-of-the-art methods. In addition, tests on the NISQA_TEST_LIVETALK real dataset of the NISQA Corpus show the breadth and robustness of our model on speech enhancement tasks.

multidisciplinary sciences
A Joint Framework of Denoising Autoencoder and Generative Vocoder for Monaural Speech Enhancement

Zhihao Du,Xueliang Zhang,Jiqing Han

DOI: https://doi.org/10.1109/taslp.2020.2991537

2020-01-01

Abstract:Conventional monaural speech enhancement methods usually enhance the magnitude spectrum of noisy speech and leave the phase unchanged. Recent studies suggest that phase is also important for both speech intelligibility and perceptual quality. Although deep learning exhibits great potential on enhancing the magnitude and phase spectra in complex spectrogram domain and waveform domain, complex spectrogram and waveform are always more difficult to predict than the magnitude spectrum due to lack of clear structure in them. In this study, a Mel-domain denoising autoencoder and a deep generative vocoder are stacked to form a joint framework for monaural speech enhancement, in which the clean speech waveform is reconstructed without using the phase. Specifically, a convolutional recurrent network (CRN) is employed as the denoising autoencoder to enhance the Mel power spectrum of noisy speech. Then, the enhanced Mel power spectrum is fed to a deep generative vocoder to synthesize the speech waveform. Furthermore, the denoising autoencoder and generative vocoder are jointly fine-tuned. Experimental results show that the proposed method significantly improves speech intelligibility and perceptual quality. More importantly, our method achieves much better generalization ability for untrained noises than previous methods.
SEGAN: Speech Enhancement Generative Adversarial Network

Santiago Pascual,A. Bonafonte,J. Serrà

DOI: https://doi.org/10.21437/Interspeech.2017-1428

2017-03-28

Abstract:Current speech enhancement techniques operate on the spectral domain and/or exploit some higher-level feature. The majority of them tackle a limited number of noise conditions and rely on first-order statistics. To circumvent these issues, deep networks are being increasingly used, thanks to their ability to learn complex functions from large example sets. In this work, we propose the use of generative adversarial networks for speech enhancement. In contrast to current techniques, we operate at the waveform level, training the model end-to-end, and incorporate 28 speakers and 40 different noise conditions into the same model, such that model parameters are shared across them. We evaluate the proposed model using an independent, unseen test set with two speakers and 20 alternative noise conditions. The enhanced samples confirm the viability of the proposed model, and both objective and subjective evaluations confirm the effectiveness of it. With that, we open the exploration of generative architectures for speech enhancement, which may progressively incorporate further speech-centric design choices to improve their performance.

Mathematics,Computer Science
A Model of Emotional Speech Generation Based on Conditional Generative Adversarial Networks

Chunjun Zheng,Wei Sun,Ning Jia

DOI: https://doi.org/10.1109/ihmsc.2019.00033

2019-08-01

Abstract:This paper presents a speech generation technology based on Conditional Generative Adversarial Networks (Conditional GAN). By introducing emotional conditions and learning emotional information in the speech database, a new voice with specified emotions can be generated independently. Generative Adversarial Networks consists of a Discriminator (D) and a Generator (G). Tensorflow is used as a learning framework, Conditional GAN model is used to train a large number of emotional voice, and voice generation network G and generation network D are used to form a dynamic “game process” to learn and observe the conditional distribution of emotional voice data better. The generated samples are close to the natural speech signals of the original learning content, which are rich in diversity and can approximate the real emotional speech. The proposed solution is evaluated on the Interactive Emotional Dyadic Motion Capture (IEMOCAP) corpus and student emotional database, and shows more accurate results than existing emotional speech generation algorithms.
Conditional Sound Generation Using Neural Discrete Time-Frequency Representation Learning

Xubo Liu,Turab Iqbal,Jinzheng Zhao,Qiushi Huang,Mark D. Plumbley,Wenwu Wang

DOI: https://doi.org/10.48550/arXiv.2107.09998

2021-10-06

Abstract:Deep generative models have recently achieved impressive performance in speech and music synthesis. However, compared to the generation of those domain-specific sounds, generating general sounds (such as siren, gunshots) has received less attention, despite their wide applications. In previous work, the SampleRNN method was considered for sound generation in the time domain. However, SampleRNN is potentially limited in capturing long-range dependencies within sounds as it only back-propagates through a limited number of samples. In this work, we propose a method for generating sounds via neural discrete time-frequency representation learning, conditioned on sound classes. This offers an advantage in efficiently modelling long-range dependencies and retaining local fine-grained structures within sound clips. We evaluate our approach on the UrbanSound8K dataset, compared to SampleRNN, with the performance metrics measuring the quality and diversity of generated sounds. Experimental results show that our method offers comparable performance in quality and significantly better performance in diversity.

Audio and Speech Processing,Artificial Intelligence,Sound
Collaborative Deep Learning for Speech Enhancement: A Run-Time Model Selection Method Using Autoencoders

Minje Kim

DOI: https://doi.org/10.48550/arXiv.1705.10385

2017-05-30

Abstract:We show that a Modular Neural Network (MNN) can combine various speech enhancement modules, each of which is a Deep Neural Network (DNN) specialized on a particular enhancement job. Differently from an ordinary ensemble technique that averages variations in models, the propose MNN selects the best module for the unseen test signal to produce a greedy ensemble. We see this as Collaborative Deep Learning (CDL), because it can reuse various already-trained DNN models without any further refining. In the proposed MNN selecting the best module during run time is challenging. To this end, we employ a speech AutoEncoder (AE) as an arbitrator, whose input and output are trained to be as similar as possible if its input is clean speech. Therefore, the AE can gauge the quality of the module-specific denoised result by seeing its AE reconstruction error, e.g. low error means that the module output is similar to clean speech. We propose an MNN structure with various modules that are specialized on dealing with a specific noise type, gender, and input Signal-to-Noise Ratio (SNR) value, and empirically prove that it almost always works better than an arbitrarily chosen DNN module and sometimes as good as an oracle result.

Sound,Machine Learning
Unsupervised speech enhancement with diffusion-based generative models

Berné Nortier,Mostafa Sadeghi,Romain Serizel

2023-09-19

Abstract:Recently, conditional score-based diffusion models have gained significant attention in the field of supervised speech enhancement, yielding state-of-the-art performance. However, these methods may face challenges when generalising to unseen conditions. To address this issue, we introduce an alternative approach that operates in an unsupervised manner, leveraging the generative power of diffusion models. Specifically, in a training phase, a clean speech prior distribution is learnt in the short-time Fourier transform (STFT) domain using score-based diffusion models, allowing it to unconditionally generate clean speech from Gaussian noise. Then, we develop a posterior sampling methodology for speech enhancement by combining the learnt clean speech prior with a noise model for speech signal inference. The noise parameters are simultaneously learnt along with clean speech estimation through an iterative expectationmaximisation (EM) approach. To the best of our knowledge, this is the first work exploring diffusion-based generative models for unsupervised speech enhancement, demonstrating promising results compared to a recent variational auto-encoder (VAE)-based unsupervised approach and a state-of-the-art diffusion-based supervised method. It thus opens a new direction for future research in unsupervised speech enhancement.

Computer Vision and Pattern Recognition,Sound,Audio and Speech Processing,Signal Processing,Machine Learning
Multi-Metric Optimization using Generative Adversarial Networks for Near-End Speech Intelligibility Enhancement

Haoyu Li,Junichi Yamagishi

DOI: https://doi.org/10.48550/arXiv.2104.08499

2021-09-16

Abstract:The intelligibility of speech severely degrades in the presence of environmental noise and reverberation. In this paper, we propose a novel deep learning based system for modifying the speech signal to increase its intelligibility under the equal-power constraint, i.e., signal power before and after modification must be the same. To achieve this, we use generative adversarial networks (GANs) to obtain time-frequency dependent amplification factors, which are then applied to the input raw speech to reallocate the speech energy. Instead of optimizing only a single, simple metric, we train a deep neural network (DNN) model to simultaneously optimize multiple advanced speech metrics, including both intelligibility- and quality-related ones, which results in notable improvements in performance and robustness. Our system can not only work in non-realtime mode for offline audio playback but also support practical real-time speech applications. Experimental results using both objective measurements and subjective listening tests indicate that the proposed system significantly outperforms state-ofthe-art baseline systems under various noisy and reverberant listening conditions.

Audio and Speech Processing,Sound

A Conditional Generative Model for Speech Enhancement

Low-latency Speech Enhancement via Speech Token Generation

Unifying Robustness and Fidelity: A Comprehensive Study of Pretrained Generative Methods for Speech Enhancement in Adverse Conditions

PAGAN: A Phase-Adapted Generative Adversarial Networks for Speech Enhancement

DCCRGAN: Deep Complex Convolution Recurrent Generator Adversarial Network for Speech Enhancement

An Experimental Study on Speech Enhancement Based on Deep Neural Networks

A regression approach to speech enhancement based on deep neural networks

Audio-Visual Speech Enhancement with Score-Based Generative Models

GAN-Based Speech Enhancement for Low SNR Using Latent Feature Conditioning

Unsupervised speech enhancement with deep dynamical generative speech and noise models

Speech enhancement using progressive learning-based convolutional recurrent neural network

Conditional Generative Adversarial Networks for Speech Enhancement and Noise-Robust Speaker Verification

Mixed T-domain and TF-domain Magnitude and Phase representations for GAN-based speech enhancement

A Joint Framework of Denoising Autoencoder and Generative Vocoder for Monaural Speech Enhancement

SEGAN: Speech Enhancement Generative Adversarial Network

A Model of Emotional Speech Generation Based on Conditional Generative Adversarial Networks

Conditional Sound Generation Using Neural Discrete Time-Frequency Representation Learning

Collaborative Deep Learning for Speech Enhancement: A Run-Time Model Selection Method Using Autoencoders

Unsupervised speech enhancement with diffusion-based generative models

Multi-Metric Optimization using Generative Adversarial Networks for Near-End Speech Intelligibility Enhancement