Abstract:Future wearable technology may provide for enhanced communication in noisy environments and for the ability to pick out a single talker of interest in a crowded room simply by the listener shifting their attentional focus. Such a system relies on two components, speaker separation and decoding the listener's attention to acoustic streams in the environment. To address the former, we present a system for joint speaker separation and noise suppression, referred to as the Binaural Enhancement via Attention Masking Network (BEAMNET). The BEAMNET system is an end-to-end neural network architecture based on self-attention. Binaural input waveforms are mapped to a joint embedding space via a learned encoder, and separate multiplicative masking mechanisms are included for noise suppression and speaker separation. Pairs of output binaural waveforms are then synthesized using learned decoders, each capturing a separated speaker while maintaining spatial cues. A key contribution of BEAMNET is that the architecture contains a separation path, an enhancement path, and an autoencoder path. This paper proposes a novel loss function which simultaneously trains these paths, so that disabling the masking mechanisms during inference causes BEAMNET to reconstruct the input speech signals. This allows dynamic control of the level of suppression applied by BEAMNET via a minimum gain level, which is not possible in other state-of-the-art approaches to end-to-end speaker separation. This paper also proposes a perceptually-motivated waveform distance measure. Using objective speech quality metrics, the proposed system is demonstrated to perform well at separating two equal-energy talkers, even in high levels of background noise. Subjective testing shows an improvement in speech intelligibility across a range of noise levels, for signals with artificially added head-related transfer functions and background noise. Finally, when used as part of an auditory attention decoder (AAD) system using existing electroencephalogram (EEG) data, BEAMNET is found to maintain the decoding accuracy achieved with ideal speaker separation, even in severe acoustic conditions. These results suggest that this enhancement system is highly effective at decoding auditory attention in realistic noise environments, and could possibly lead to improved speech perception in a cognitively controlled hearing aid.

Cognitive-Driven Binaural Beamforming Using EEG-Based Auditory Attention Decoding

Cognitive-driven convolutional beamforming using EEG-based auditory attention decoding

Speaker separation in realistic noise environments with applications to a cognitively-controlled hearing aid

Neural decoding of attentional selection in multi-speaker environments without access to clean sources

Linear versus deep learning methods for noisy speech separation for EEG-informed attention decoding

Real-time control of a hearing instrument with EEG-based attention decoding

Improved Decoding of Attentional Selection in Multi-Talker Environments with Self-Supervised Learned Speech Representation

Improving auditory attention decoding performance of linear and non-linear methods using state-space model

Brain-controlled augmented hearing for spatially moving conversations in multi-talker environments

Subspace Hybrid MVDR Beamforming for Augmented Hearing

A Speech Enhancement Method Combining Beamforming with RNN for Hearing Aids.

EEG-informed attended speaker extraction from recorded speech mixtures with application in neuro-steered hearing prostheses

EEG-based auditory attention decoding with audiovisual speech for hearing-impaired listeners

Adaptive Dereverberation, Noise and Interferer Reduction Using Sparse Weighted Linearly Constrained Minimum Power Beamforming

Fast EEG-Based Decoding Of The Directional Focus Of Auditory Attention Using Common Spatial Patterns

Congruent audiovisual speech enhances auditory attention decoding with EEG.

Comparison of linear and nonlinear methods for decoding selective attention to speech from ear-EEG recordings

An Interpretable Performance Metric for Auditory Attention Decoding Algorithms in a Context of Neuro-Steered Gain Control

Benefits of triple acoustic beamforming during speech-on-speech masking and sound localization for bilateral cochlear-implant users

Real-Time Tracking of Selective Auditory Attention From M/EEG: A Bayesian Filtering Approach

Deep learning-based auditory attention decoding in listeners with hearing impairment