Exploiting Consistency-Preserving Loss and Perceptual Contrast Stretching to Boost SSL-based Speech Enhancement

Muhammad Salman Khan,Moreno La Quatra,Kuo-Hsuan Hung,Szu-Wei Fu,Sabato Marco Siniscalchi,Yu Tsao

2024-08-09

Abstract:Self-supervised representation learning (SSL) has attained SOTA results on several downstream speech tasks, but SSL-based speech enhancement (SE) solutions still lag behind. To address this issue, we exploit three main ideas: (i) Transformer-based masking generation, (ii) consistency-preserving loss, and (iii) perceptual contrast stretching (PCS). In detail, conformer layers, leveraging an attention mechanism, are introduced to effectively model frame-level representations and obtain the Ideal Ratio Mask (IRM) for SE. Moreover, we incorporate consistency in the loss function, which processes the input to account for the inconsistency effects of signal reconstruction from the spectrogram. Finally, PCS is employed to improve the contrast of input and target features according to perceptual importance. Evaluated on the VoiceBank-DEMAND task, the proposed solution outperforms previously SSL-based SE solutions when tested on several objective metrics, attaining a SOTA PESQ score of 3.54.

Sound,Audio and Speech Processing

What problem does this paper attempt to address?

This paper attempts to solve the problem that self - supervised learning (SSL) performs less well in the speech enhancement (SE) task than other downstream speech tasks. Specifically, although SSL has achieved state - of - the - art results in multiple downstream speech tasks, there are still deficiencies in SSL - based speech enhancement solutions. To narrow the gap with state - of - the - art speech enhancement methods, the author proposes the following three main improvement measures:

1. **Transformer - based Mask Generation**: Use Conformer layers (which combine the advantages of convolution and Transformer), and by introducing the attention mechanism, effectively model frame - level representations and obtain the ideal ratio mask (IRM) for speech enhancement.
2. **Consistency - Preserving Loss**: Introduce consistency in the loss function to deal with the inconsistent effects that occur when reconstructing signals from spectrograms. This helps improve the model's ability to process the input signal consistently.
3. **Perceptual Contrast Stretching (PCS)**: Improve the contrast between input and target features by making perceptual importance adjustments. PCS, as a pre - processing step, is applied to noise and target waveforms before training to better utilize the sensitivity of the human auditory system.

These methods work together to aim at improving the performance of SSL - based speech enhancement models and have achieved a state - of - the - art PESQ score of 3.54 on the VoiceBank - DEMAND task.