Abstract:Earphones have become a popular voice input and interaction device. However, airborne speech is susceptible to ambient noise, making it necessary to improve the quality and intelligibility of speech on earphones in noisy conditions. As the dual-microphone structure (i.e., outer and in-ear microphones) has been widely adopted in earphones (especially ANC earphones), we design EarSpeech which exploits in-ear acoustic sensory as the complementary modality to enable airborne speech enhancement. The key idea of EarSpeech is that in-ear speech is less sensitive to ambient noise and exhibits a correlation with airborne speech. However, due to the occlusion effect, in-ear speech has limited bandwidth, making it challenging to directly correlate with full-band airborne speech. Therefore, we exploit the occlusion effect to carry out theoretical modeling and quantitative analysis of this cross-channel correlation and study how to leverage such cross-channel correlation for speech enhancement. Specifically, we design a series of methodologies including data augmentation, deep learning-based fusion, and noise mixture scheme, to improve the generalization, effectiveness, and robustness of EarSpeech, respectively. Lastly, we conduct real-world experiments to evaluate the performance of our system. Specifically, EarSpeech achieves an average improvement ratio of 27.23% and 13.92% in terms of PESQ and STOI, respectively, and significantly improves SI-SDR by 8.91 dB. Benefiting from data augmentation, EarSpeech can achieve comparable performance with a small-scale dataset that is 40 times less than the original dataset. In addition, we validate the generalization of different users, speech content, and language types, respectively, as well as robustness in the real world via comprehensive experiments. The audio demo of EarSpeech is available on https://github.com/EarSpeech/earspeech.github.io/.

Domain-adaptation method between acoustic-response data using different insert earphones

Stethoscope-guided Supervised Contrastive Learning for Cross-domain Adaptation on Respiratory Sound Classification

EarDA: Towards Accurate and Data-Efficient Earable Activity Sensing

The Application Design of Hearing Aid Parameters Auto Adaptive System for Hearing Impaired Children Based on Android Terminal

Unsupervised Domain Adaptation for Acoustic Scene Classification Using Band-Wise Statistics Matching

Domain adaptation for ear recognition using deep convolutional neural networks

Domain Adaptation for Epileptic EEG Classification Using Adversarial Learning and Riemannian Manifold

Deep Neural Network Based Environment Sound Classification and Its Implementation on Hearing Aid App

Mind the Domain Gap: a Systematic Analysis on Bioacoustic Sound Event Detection

Domain Adaptation with Augmented Data by Deep Neural Network Based Method Using Re-Recorded Speech for Automatic Speech Recognition in Real Environment

EarSpeech: Exploring In-Ear Occlusion Effect on Earphones for Data-efficient Airborne Speech Enhancement

Ambulatory Phonation Monitoring Using Wireless Bluetooth Earphones

Learning domain-invariant classifiers for infant cry sounds

Disentangled and Side-Aware Unsupervised Domain Adaptation for Cross-Dataset Subjective Tinnitus Diagnosis

Improving EEG-based decoding of the locus of auditory attention through domain adaptation

Domain Information Control at Inference Time for Acoustic Scene Classification

Multi-source Domain Adaptation for Text-independent Forensic Speaker Recognition

A Study of Enhancement, Augmentation, and Autoencoder Methods for Domain Adaptation in Distant Speech Recognition

A Contrastive Embedding-Based Domain Adaptation Method for Lung Sound Recognition in Children Community-Acquired Pneumonia

Disentangled Adversarial Domain Adaptation for Phonation Mode Detection in Singing and Speech

Prototypical Contrastive Domain Adaptation Network for Nonstationary EEG Classification