Abstract:Objective: People who suffer from hearing impairments can find it difficult to follow a conversation in a multi-speaker environment. Current hearing aids can suppress background noise; however, there is little that can be done to help a user attend to a single conversation amongst many without knowing which speaker the user is attending to. Cognitively controlled hearing aids that use auditory attention decoding (AAD) methods are the next step in offering help. Translating the successes in AAD research to real-world applications poses a number of challenges, including the lack of access to the clean sound sources in the environment with which to compare with the neural signals. We propose a novel framework that combines single-channel speech separation algorithms with AAD. Approach: We present an end-to-end system that (1) receives a single audio channel containing a mixture of speakers that is heard by a listener along with the listener's neural signals, (2) automatically separates the individual speakers in the mixture, (3) determines the attended speaker, and (4) amplifies the attended speaker's voice to assist the listener. Main results: Using invasive electrophysiology recordings, we identified the regions of the auditory cortex that contribute to AAD. Given appropriate electrode locations, our system is able to decode the attention of subjects and amplify the attended speaker using only the mixed audio. Our quality assessment of the modified audio demonstrates a significant improvement in both subjective and objective speech quality measures. Significance: Our novel framework for AAD bridges the gap between the most recent advancements in speech processing technologies and speech prosthesis research and moves us closer to the development of cognitively controlled hearable devices for the hearing impaired.

NeuroHeed+: Improving Neuro-steered Speaker Extraction with Joint Auditory Attention Detection

NeuroHeed: Neuro-Steered Speaker Extraction using EEG Signals

NeuroSpex: Neuro-Guided Speaker Extraction with Cross-Modal Attention

EEG-informed attended speaker extraction from recorded speech mixtures with application in neuro-steered hearing prostheses

Extracting the Auditory Attention in a Dual-Speaker Scenario From EEG Using a Joint CNN-LSTM Model

EEG-based Auditory Attention Decoding: Towards Neuro-Steered Hearing Devices

Neural decoding of attentional selection in multi-speaker environments without access to clean sources

EEG-based detection of the locus of auditory attention with convolutional neural networks

Improved Decoding of Attentional Selection in Multi-Talker Environments with Self-Supervised Learned Speech Representation

Auditory attention decoding from electroencephalography based on long short-term memory networks

EEG-Based Short-Time Auditory Attention Detection Using Multi-Task Deep Learning.

Corticomorphic Hybrid CNN-SNN Architecture for EEG-based Low-footprint Low-latency Auditory Attention Detection

MSFNet: Multi-Scale Fusion Network for Brain-Controlled Speaker Extraction

BASEN: Time-Domain Brain-Assisted Speech Enhancement Network with Convolutional Cross Attention in Multi-talker Conditions

Linear versus deep learning methods for noisy speech separation for EEG-informed attention decoding

A Multi-Plane Decoupled Convolutional Network for EEG-Based Auditory Attention Detection

Cognitive-driven convolutional beamforming using EEG-based auditory attention decoding

Brain-controlled augmented hearing for spatially moving conversations in multi-talker environments

EEG-Based Auditory Attention Detection via Frequency and Channel Neural Attention

Using Ear-EEG to Decode Auditory Attention in Multiple-speaker Environment

Real-time control of a hearing instrument with EEG-based attention decoding