Abstract:Individuals with hearing impairments face challenges in their ability to comprehend speech, particularly in noisy environments. The aim of this study is to explore the effectiveness of audio-visual speech enhancement (AVSE) in enhancing the intelligibility of vocoded speech in cochlear implant (CI) simulations. Notably, the study focuses on a challenged scenario where there is limited availability of training data for the AVSE task. To address this problem, we propose a novel deep neural network framework termed Self-Supervised Learning-based AVSE (SSL-AVSE). The proposed SSL-AVSE combines visual cues, such as lip and mouth movements, from the target speakers with corresponding audio signals. The contextually combined audio and visual data are then fed into a Transformer-based SSL AV-HuBERT model to extract features, which are further processed using a BLSTM-based SE model. The results demonstrate several key findings. Firstly, SSL-AVSE successfully overcomes the issue of limited data by leveraging the AV-HuBERT model. Secondly, by fine-tuning the AV-HuBERT model parameters for the target SE task, significant performance improvements are achieved. Specifically, there is a notable enhancement in PESQ (Perceptual Evaluation of Speech Quality) from 1.43 to 1.67 and in STOI (Short-Time Objective Intelligibility) from 0.70 to 0.74. Furthermore, the performance of the SSL-AVSE was evaluated using CI vocoded speech to assess the intelligibility for CI users. Comparative experimental outcomes reveal that in the presence of dynamic noises encountered during human conversations, SSL-AVSE exhibits a substantial improvement. The NCM (Normal Correlation Matrix) values indicate an increase of 26.5% to 87.2% compared to the noisy baseline.

A Supervised Speech Enhancement Method for Smartphone-Based Binaural Hearing Aids

A Smart Binaural Hearing Aid Architecture Leveraging a Smartphone APP with Deep-Learning Speech Enhancement.

An RNN-based Speech Enhancement Method for a Binaural Hearing Aid System

Deep Neural Network Based Noised Asian Speech Enhancement and Its Implementation on a Hearing Aid App.

A MVDR- MWF Combined Algorithm for Binaural Hearing Aid System

A Speech Enhancement Method Combining Beamforming with RNN for Hearing Aids.

A Lightweight and Real-Time Binaural Speech Enhancement Model with Spatial Cues Preservation

Real-time multichannel deep speech enhancement in hearing aids: Comparing monaural and binaural processing in complex acoustic scenarios

A Smart Binaural Hearing Aid Architecture Based on a Mobile Computing Platform

A non-invasive speech quality evaluation algorithm for hearing aids with multi-head self-attention and audiogram-based features

System Architecture of a Smart Binaural Hearing Aid Using a Mobile Computing Platform

Deep Neural Network Based Environment Sound Classification and Its Implementation on Hearing Aid App

Deep Learning Based Real-Time Speech Enhancement for Dual-Microphone Mobile Phones

Electroacoustic Evaluation of Smartphone-Based Hearing Aid Applications

A Real-Time Tone Enhancement Method for Continuous Mandarin Speeches

Improving Deep Neural Network Based Speech Enhancement in Low SNR Environments

Densely Connected Multi-Stage Model with Channel Wise Subband Feature for Real-Time Speech Enhancement.

Deep learning-based denoising streamed from mobile phones improves speech-in-noise understanding for hearing aid users

Audio-Visual Speech Enhancement Using Self-supervised Learning to Improve Speech Intelligibility in Cochlear Implant Simulations

Restoring speech intelligibility for hearing aid users with deep learning

A Refining Underlying Information Framework for Monaural Speech Enhancement