Abstract:Silent Speech Interface (SSI) have been developed to convert silent articulatory gestures into speech, facilitating silent speech in public spaces and aiding individuals with aphasia. Prior arts of SSI, either relying on wearable devices or cameras, may lead to extended contact requirements or privacy leakage risks. Recent advancements in acoustic sensing offer new opportunitis for gesture sensing. However, they typically focus on content classification rather than on reconstructing audible speech, leading to the loss of crucial speech characteristics such as speech rate, intonation, and emotion. In this paper, we propose UltraSR, a novel sensing system that supports accurate audible speech reconstruction by analyzing the disturbance of tiny articulatory gestures on the reflected ultrasound signal. The design of UltraSR introduces a multi-scale feature extraction scheme for aggregating information from multiple views, and a new model that provides the unique mapping relationship between ultrasound and speech signals, so that the audible speech can be successfully reconstructed from the silent speech. However, establishing the mapping relationship depends on plenty of training data. Instead of the time-consuming collection of massive amounts of data for training, we construct an inverse task that constitutes a dual form with the original task to generate virtual gestures from widely available audio (e.g., phone calls) for facilitating model training. Furthermore, we introduce a fine-tuning mechanism using unlabeled data for user adaptation. We implement UltraSR using a portable smartphone and evaluate it in various environments. The evaluation results show that UltraSR can reconstruct speech with a (Character Error Rate) CER as low as 5.22%, and decrease the CER from 80.13% to 6.31% on new users with only 1 hour of ultrasound signals provided, which outperforms state-of-the-art acoustic-based approaches while preserving rich speech information.

Scalable Identity-Oriented Speech Retrieval

Speech Personality Recognition Based on Annotation Classification Using Log-Likelihood Distance and Extraction of Essential Audio Features.

Seeing Your Speech Style: A Novel Zero-Shot Identity-Disentanglement Face-based Voice Conversion

Speaker-Independent Speech Emotion Recognition Based On Cnn-Blstm And Multiple Svms

SVoice: Enabling Voice Communication in Silence Via Acoustic Sensing on Commodity Devices.

SVoice

A Speech Indexing System For Recorded Audio Source Based On Speaker Identification Technique

UltraSR: Silent Speech Reconstruction Via Acoustic Sensing

SAIC: Integration of Speech Anonymization and Identity Classification

Towards Universal Speech Discrete Tokens: A Case Study for ASR and TTS

Enhancing Open-Set Speaker Identification through Rapid Tuning with Speaker Reciprocal Points and Negative Sample

Query Expansion Based High Performance Chinese Voice Retrieval

Enhancing Query Expansion for Semantic Retrieval of Spoken Content with Automatically Discovered Acoustic Patterns.

Scalable Face Image Retrieval with Identity-Based Quantization and Multireference Reranking

OpenVoice: Versatile Instant Voice Cloning

The SpeakIn System Description for CNSRC2022

Beyond Performance Plateaus: A Comprehensive Study on Scalability in Speech Enhancement

Towards Vocabulary-Independent Speech Indexing for Large-Scale Repositories

Exploring Speech Recognition, Translation, and Understanding with Discrete Speech Units: A Comparative Study

Speech Retrieval Engine Based on Relevance Feedback

SpeechQoE