Abstract:In real-world scenarios, dynamic ambient noise often degrades speech quality, highlighting the need for advanced speech enhancement techniques. Traditional methods, which rely on static embeddings as auxiliary features, struggle to address the complexities of varying noise conditions. To overcome this, we propose a Dual-stream Noise and Speech Information Perception (DNSIP) approach that dynamically detects and processes both noise and speech through innovative information extraction and suppression mechanisms. Initially, non-speech segments predominantly contain environmental noise, while speech segments carry information about the intended speaker. To handle this dynamic nature, real-time voice activity detection (VAD) is employed to accurately differentiate between speech and noise components. Building on VAD estimates, we propose an innovative information extraction framework that selectively extracts relevant noise and speech features from the noisy input, establishing a dual-stream network for concurrent noise and speech learning. To account for the temporal and spectral variability of noise and speech, a frequency-sequence attention mechanism is integrated, enhancing the model's ability to learn contextual and spectral dependencies. Additionally, an information suppression module is introduced to minimize cross-stream interference by attenuating noise within the speech stream and suppressing speech content within the noise stream. The derived noise and speech spectrograms are then utilized to formulate a minimum mean square error log-spectral amplitude (MMSE-LSA) estimator for robust speech enhancement. Experimental evaluations on the WSJ0 and VCTK+DEMAND datasets demonstrate that our DNSIP approach surpasses existing state-of-the-art methods, underscoring its efficacy in challenging acoustic environments.

Speech Enhancement Based on Multi-Stream Model

Densely Connected Multi-Stage Model with Channel Wise Subband Feature for Real-Time Speech Enhancement.

Two-Stage Single-Channel Speech Enhancement with Multi-Frame Filtering

FB-MSTCN: A Full-Band Single-Channel Speech Enhancement Method Based on Multi-Scale Temporal Convolutional Network

Multi-stage Strength Estimation Network with Cross Attention for Single Channel Speech Enhancement

Single-Channel Speech Enhancement Based on Improved Frame-Iterative Spectral Subtraction in the Modulation Domain

Exploring Conventional Enhancement and Separation Methods for Multi‐speech Enhancement in Indoor Environments

A Speech Enhancement Algorithm By Iterating Single- And Multi-Microphone Processing And Its Application To Robust Asr

Single-channel Speech Enhancement Student under Multi-channel Speech Enhancement Teacher

An End-to-End Speech Enhancement Framework Using Stacked Multi-scale Blocks.

Dual-stream Noise and Speech Information Perception based Speech Enhancement

Multi-channel speech enhancement using sparse coding on local time-frequency structures.

Stream Attention For Distributed Multi-Microphone Speech Recognition

Speech Enhancement Based on the Multi-Scales and Multi-Thresholds of the Auditory Perception Wavelet Transform

Speech enhancement based on spectral subtraction of multi-band scale

Speech Enhancement for Nonstationary Noise Environments

Speech Enhancement Based on Beamforming and Post-Filtering by Combining Phase Information

Dbn Based Multi-Stream Models For Speech

Two-stage unet with channel and temporal-frequency attention for multi-channel speech enhancement

Improved Codebook-Based Speech Enhancement Based On Mbe Model

Two Heads Are Better Than One: A Two-Stage Complex Spectral Mapping Approach for Monaural Speech Enhancement.