Thunder : Unified Regression-Diffusion Speech Enhancement with a Single Reverse Step using Brownian Bridge

Thanapat Trachu,Chawan Piansaddhayanon,Ekapol Chuangsuwanich

2024-06-10

Abstract:Diffusion-based speech enhancement has shown promising results, but can suffer from a slower inference time. Initializing the diffusion process with the enhanced audio generated by a regression-based model can be used to reduce the computational steps required. However, these approaches often necessitate a regression model, further increasing the system's complexity. We propose Thunder, a unified regression-diffusion model that utilizes the Brownian bridge process which can allow the model to act in both modes. The regression mode can be accessed by setting the diffusion time step closed to 1. However, the standard score-based diffusion modeling does not perform well in this setup due to gradient instability. To mitigate this problem, we modify the diffusion model to predict the clean speech instead of the score function, achieving competitive performance with a more compact model size and fewer reverse steps.

Sound,Artificial Intelligence,Computation and Language,Audio and Speech Processing

What problem does this paper attempt to address?

The main problem that this paper attempts to solve is that the existing speech enhancement methods based on diffusion models have the problem of overly long inference time in practical applications. Specifically:

1. **Slow Inference Speed of Diffusion Models**: Traditional diffusion models require multiple reverse diffusion steps to gradually denoise, which leads to a long inference time and limits their use in real - time applications.
2. **Complexity of the Regression - Diffusion Two - Stage Model**: Some methods (such as StoRM) reduce the required diffusion steps by combining regression models and diffusion models, but this increases the complexity and the number of parameters of the system.

To solve these problems, the paper proposes a unified regression - diffusion model - Thunder, which utilizes the Brownian Bridge Process to enable the model to switch between regression mode and diffusion mode, thereby achieving more efficient speech enhancement. The main improvement points include:

- **Single - Step Prediction Ability**: By setting the diffusion time step close to 1, Thunder can perform single - step prediction in the regression mode, significantly reducing the inference time.
- **Simplified Model Structure**: Thunder integrates regression and diffusion functions into one model, avoiding additional parameter increases while maintaining competitive performance.

### Specific Problem Summary

The paper aims to solve the following specific problems:
- **Reducing Inference Time**: By introducing the Brownian Bridge Process and the single - step prediction mechanism, reduce the reverse steps required by the diffusion model.
- **Simplifying System Complexity**: Integrate regression and diffusion functions into one model, avoiding the complexity and parameter redundancy brought by multi - model combinations.
- **Improving Model Efficiency**: On the premise of maintaining competitive performance, reduce the number of model parameters and improve computational efficiency.

Through these improvements, Thunder not only achieves results comparable to or better than existing methods on standard data sets, but also shows good generalization ability in cross - domain tests.