Abstract:Silent Speech Interface (SSI) has been proposed as a means of reconstructing audible speech from silent articulatory gestures for covert voice communication in public and voice assistance for the aphasic. Prior arts of SSI, either relying on wearable devices or cameras, may lead to extended contact requirements or privacy leakage risks. The recent advances in acoustic sensing have brought new opportunities for sensing gestures, but their original intention is to infer speech content for classification instead of audible speech reconstruction, resulting in the loss of some important speech information (e.g., speech rate, intonation, and emotion). In this paper, we propose, the first system that supports accurate audible speech reconstruction by analyzing the disturbance of tiny articulatory gestures on the reflected ultrasound signal. The design of introduces a new model that provides the unique mapping relationship between ultrasound and speech signals, so that the audible speech can be successfully reconstructed from the silent speech. However, establishing the mapping relationship depends on plenty of training data. Instead of the time-consuming collection of massive amounts of data for training, we construct an inverse task that constitutes a dual form with the original task to generate virtual gestures from widely available audio (e.g., phone calls) for facilitating model training. Furthermore, we introduce a fine-tuning mechanism using unlabeled data for user adaptation. We implement using a portable smartphone and evaluate it in various environments. The evaluation results show that can reconstruct speech with a (Character Error Rate) CER as low as 7.62%, and decrease the CER from 82.77% to 9.42% on new users with only 1 hour of ultrasound signals provided, which outperforms state-of-the-art acoustic-based approaches while preserving rich speech information.

Voice Recovery from Human Surroundings with Millimeter Wave Radar

Speech Acquisition and Recovery Based on Piezoelectric Effect in the Mmwave Band

A Novel and Efficient Voice Activity Detector Using Shape Features of Speech Wave.

Wavoice: an Mmwave-Assisted Noise-Resistant Speech Recognition System

Wavoice: an Mmwave-Assisted Noise-Resistant Speech Recognition System.

Wavoice: A mmWave-assisted Noise-resistant Speech Recognition SystemJust Accepted

Detection Of Bio-Signals From Body Movement Based On High-Dynamic-Range Doppler Radar Sensor (Invited)

AmbiEar: mmWave Based Voice Recognition in NLoS Scenarios

Mmphone: Sound Recovery Using Millimeter-Wave Radios with Adaptive Fusion Enhanced Vibration Sensing

mmSafe: A Voice Security Verification System Based on Millimeter-Wave Radar

Doppler Vital Signs Detection in the Presence of Large-Scale Random Body Movements

Mmeavesdropper: Signal Augmentation-based Directional Eavesdropping with Mmwave Radar.

SVoice

Blind Separation of Doppler Human Gesture Signals Based on Continuous-Wave Radar Sensors

SuperVoice: Text-Independent Speaker Verification Using Ultrasound Energy in Human Speech

Poster Abstract: Integrated Sensing and Communication between Daily Devices and mmWave Radars

SVoice: Enabling Voice Communication in Silence Via Acoustic Sensing on Commodity Devices.

Deep Learning Radar for High Fidelity Heart Sound Recovery in Real-World Scenarios

Mmmic: Multi-modal Speech Recognition Based on Mmwave Radar.

A Novel Method for Speech Acquisition and Enhancement by 94 GHz Millimeter-Wave Sensor

RFMic-Phone: Robust Sound Acquisition Combining Millimeter-Wave Radar and Microphone