Abstract:The recent development of speech enhancement methods has incorporated attention mechanisms for learning long-term speech signal dependencies. The utilization of deep convolution networks (DCN) equipped with the self-attention (SA) and transformers has showed promising results in speech enhancement (SE). While self-attention networks excel in extracting significant long-sequence contextual information in mining tasks, they may not effectively concentrate on subtle aspects within speech signals. These subtle details include temporal or spectral continuity, spectral structure, and timbre. To tackle this problem, in the current work, we propose a novel speech enhancement model based on adaptive attention. The proposed model incorporates both local and global attention modules in between a convolutional encoder and a convolutional decoder. The local attention module (LAM) integrates channel and spatial attentions, which can make the model pay more attention to the local details in the speech block, specifically the frame-level features. And the features at utterance-level are explored through a self-attention mechanism in global attention module (GAM). Different from existing transformers, the feed forward network of GAM is improved by introducing a 1D-Conv layer and Bi-directional long short-term memory (Bi-LSTM) for extracting global features, so that the network can more effectively model long sequence context. Moreover, a CNN module is also added to global attention module so that short-term noise can be reduced more effectively, based on the ability of CNN to extract local information. The proposed model stands apart from the current speech enhancement techniques that solely rely on self-attention networks. Instead, our approach models the speech signal using two different attention networks simultaneously, both local detail information and global contextual information of speech are considered, thus better extracting useful information from the speech signal. The effectiveness of the proposed model is assessed using both objective (PESQ and STOI) and subjective tests (signal distortion (CSIG), background distortion (CBAK) and overall quality (COVL)) on two distinct datasets: Voice Bank-Demand dataset and LibriSpeech dataset. The experimental findings demonstrate that our model outperformed the competing baselines on both the datasets.

Interpretable Filter Learning Using Soft Self-attention For Raw Waveform Speech Recognition

Multi-Span Acoustic Modelling Using Raw Waveform Signals.

Learning the Front-End Speech Feature with Raw Waveform for End-to-End Speaker Recognition

Learning Filter Banks Using Deep Learning For Acoustic Signals

Self-Convolution for Automatic Speech Recognition.

Self-Attention for Audio Super-Resolution

Toward end-to-end interpretable convolutional neural networks for waveform signals

Learning neural audio features without supervision

Adaptive attention mechanism for single channel speech enhancement

Resource aware design of a deep convolutional-recurrent neural network for speech recognition through audio-visual sensor fusion

Basic Filters for Convolutional Neural Networks Applied to Music: Training or Design?

Learning Speech Representations from Raw Audio by Joint Audiovisual Self-Supervision

MuSLCAT: Multi-Scale Multi-Level Convolutional Attention Transformer for Discriminative Music Modeling on Raw Waveforms

PF-Net: Personalized Filter for Speaker Recognition from Raw Waveform

Automatic Lip-Reading with Hierarchical Pyramidal Convolution and Self-Attention for Image Sequences with No Word Boundaries

Fitting Auditory Filterbanks with Multiresolution Neural Networks

Multi-Frequency Information Enhanced Channel Attention Module for Speaker Representation Learning

Exploring spectro-temporal features in end-to-end convolutional neural networks

On Architectures and Training for Raw Waveform Feature Extraction in ASR

Lip-Reading Driven Deep Learning Approach for Speech Enhancement

Self-consistent context aware conformer transducer for speech recognition