Abstract:Objective: People who suffer from hearing impairments can find it difficult to follow a conversation in a multi-speaker environment. Current hearing aids can suppress background noise; however, there is little that can be done to help a user attend to a single conversation amongst many without knowing which speaker the user is attending to. Cognitively controlled hearing aids that use auditory attention decoding (AAD) methods are the next step in offering help. Translating the successes in AAD research to real-world applications poses a number of challenges, including the lack of access to the clean sound sources in the environment with which to compare with the neural signals. We propose a novel framework that combines single-channel speech separation algorithms with AAD. Approach: We present an end-to-end system that (1) receives a single audio channel containing a mixture of speakers that is heard by a listener along with the listener's neural signals, (2) automatically separates the individual speakers in the mixture, (3) determines the attended speaker, and (4) amplifies the attended speaker's voice to assist the listener. Main results: Using invasive electrophysiology recordings, we identified the regions of the auditory cortex that contribute to AAD. Given appropriate electrode locations, our system is able to decode the attention of subjects and amplify the attended speaker using only the mixed audio. Our quality assessment of the modified audio demonstrates a significant improvement in both subjective and objective speech quality measures. Significance: Our novel framework for AAD bridges the gap between the most recent advancements in speech processing technologies and speech prosthesis research and moves us closer to the development of cognitively controlled hearable devices for the hearing impaired.

Poster abstract: Speaker localization and cancellation for improving social attention

A Real-Time Selective Speaker Cancellation System for Relieving Social Anxiety in Autistics.

A Wearable Vision-To-Audio Sensory Substitution Device for Blind Assistance and the Correlated Neural Substrates

Poster: Inaudible High-throughput Communication Through Acoustic Signals

Brain-controlled augmented hearing for spatially moving conversations in multi-talker environments

Eye contact reminder system for people with autism

Egocentric Auditory Attention Localization in Conversations

Improved Decoding of Attentional Selection in Multi-Talker Environments with Self-Supervised Learned Speech Representation

Is Someone Speaking? Exploring Long-term Temporal Features for Audio-visual Active Speaker Detection

Real-time control of a hearing instrument with EEG-based attention decoding

Neural decoding of attentional selection in multi-speaker environments without access to clean sources

A Real-time Speaker Diarization System Based on Spatial Spectrum

Handtalker: A Multimodal Dialog System Using Sign Language and 3-D Virtual Human

Egocentric Deep Multi-Channel Audio-Visual Active Speaker Localization

An Advanced Learning Environment Aided by Recognition of Multi-modal Social Signals

Concurrent talking in immersive virtual reality: on the dominance of visual speech cues

Listen to the Speaker in Your Gaze

A System Integrating Speech Interaction and Vision Sensing Applying in Smart Home Scenario.

NeuroHeed+: Improving Neuro-steered Speaker Extraction with Joint Auditory Attention Detection

Healthcare headset with tuneable auditory characteristics control for children with Autism spectrum disorder

StereoPilot: A Wearable Target Location System for Blind and Visually Impaired Using Spatial Audio Rendering