Abstract:Future wearable technology may provide for enhanced communication in noisy environments and for the ability to pick out a single talker of interest in a crowded room simply by the listener shifting their attentional focus. Such a system relies on two components, speaker separation and decoding the listener's attention to acoustic streams in the environment. To address the former, we present a system for joint speaker separation and noise suppression, referred to as the Binaural Enhancement via Attention Masking Network (BEAMNET). The BEAMNET system is an end-to-end neural network architecture based on self-attention. Binaural input waveforms are mapped to a joint embedding space via a learned encoder, and separate multiplicative masking mechanisms are included for noise suppression and speaker separation. Pairs of output binaural waveforms are then synthesized using learned decoders, each capturing a separated speaker while maintaining spatial cues. A key contribution of BEAMNET is that the architecture contains a separation path, an enhancement path, and an autoencoder path. This paper proposes a novel loss function which simultaneously trains these paths, so that disabling the masking mechanisms during inference causes BEAMNET to reconstruct the input speech signals. This allows dynamic control of the level of suppression applied by BEAMNET via a minimum gain level, which is not possible in other state-of-the-art approaches to end-to-end speaker separation. This paper also proposes a perceptually-motivated waveform distance measure. Using objective speech quality metrics, the proposed system is demonstrated to perform well at separating two equal-energy talkers, even in high levels of background noise. Subjective testing shows an improvement in speech intelligibility across a range of noise levels, for signals with artificially added head-related transfer functions and background noise. Finally, when used as part of an auditory attention decoder (AAD) system using existing electroencephalogram (EEG) data, BEAMNET is found to maintain the decoding accuracy achieved with ideal speaker separation, even in severe acoustic conditions. These results suggest that this enhancement system is highly effective at decoding auditory attention in realistic noise environments, and could possibly lead to improved speech perception in a cognitively controlled hearing aid.

Binaural multichannel blind speaker separation with a causal low-latency and low-complexity approach

Low bit rate binaural link for improved ultra low-latency low-complexity multichannel speech enhancement in Hearing Aids

A Lightweight and Real-Time Binaural Speech Enhancement Model with Spatial Cues Preservation

Binaural Scene Analysis with Multidimensional Statistical Filters

Multichannel blind speech source separation with a disjoint constraint source model

A Separation and Interaction Framework for Causal Multi-Channel Speech Enhancement.

Single-channel Blind Separation of Co-Frequency Signals Based on Convolutional Network

Real-time binaural speech separation with preserved spatial cues

Glmsnet: single channel speech separation framework in noisy and reverberant environments

Determined Multichannel Blind Source Separation with Clustered Source Model

Multi-Channel Masking with Learnable Filterbank for Sound Source Separation

A Multichannel Learning-Based Approach for Sound Source Separation in Reverberant Environments

CrossNet: Leveraging Global, Cross-Band, Narrow-Band, and Positional Encoding for Single- and Multi-Channel Speaker Separation

SpatialNet: Extensively Learning Spatial Information for Multichannel Joint Speech Separation, Denoising and Dereverberation

Blind separation of speech sources in multichannel compressed sensing

Blind Multichannel Identification for Speech Dereverberation and Enhancement

Speaker separation in realistic noise environments with applications to a cognitively-controlled hearing aid

Towards Real-Time Single-Channel Speech Separation in Noisy and Reverberant Environments

Two-stage Blind Deconvolution Using State-space Models.

Blind Source Separation Based on NNAE-GCC.

Temporal-Spatial Neural Filter: Direction Informed End-to-End Multi-channel Target Speech Separation