Abstract:With the increasing popularity of the Internet of Things and smart factories, robust sound acquisition plays an important role in communication and human–machine interaction, for example, in-vehicle voice control interactions, smart home, and robotic voice interaction. However, due to the inherent sensing manners of the widely used microphone system, the existing audio-only sound acquisition system has fundamental limitations in solving the cocktail problem and is difficult to adapt to complex acoustic environments. Considering the advantages of millimeter-wave (mmWave) radar in the field of target positioning and precise vibration measurement, the mmWave vibration measurement-based sound perception has great potential to solve the above problems. However, there are still difficulties in high-quality sound recovery. In this letter, we propose RFMic-Phone, a robust sound acquisition system combining a mmWave radar and a traditional microphone, which combines the sound signals captured by the mmWave radar and microphone, offering a novel approach for reliable sound acquisition in the complex acoustic environment. The microphone is used to obtain the mixed and high-fidelity audio signals, while the radar is used to obtain the vibration of the sound source. To achieve intelligent and effective feature fusion, we employ a deep learning framework with a modified convolutional encoder–decoder neural network structure. Moreover, we propose to utilize the real and imaginary spectra of the target sound source as the input of the network, allowing achieve high-quality target sound signals. The experimental results show that the RFMic-Phone can achieve robust and high-quality sound signals acquisition in a variety of complex acoustic environments.

Mmmic: Multi-modal Speech Recognition Based on Mmwave Radar.

Wavoice: A mmWave-assisted Noise-resistant Speech Recognition SystemJust Accepted

Wavoice: an Mmwave-Assisted Noise-Resistant Speech Recognition System

Wavoice: an Mmwave-Assisted Noise-Resistant Speech Recognition System.

Wavoice: A Noise-resistant Multi-modal Speech Recognition System Fusing mmWave and Audio Signals

AmbiEar: mmWave Based Voice Recognition in NLoS Scenarios

Multi-target Time-Varying Vocal Folds Vibration Detection Using MIMO FMCW Radar

Robust Dual-Modal Speech Keyword Spotting for XR Headsets

RFMic-Phone: Robust Sound Acquisition Combining Millimeter-Wave Radar and Microphone

A Novel Method for Speech Acquisition and Enhancement by 94 GHz Millimeter-Wave Sensor

mmSafe: A Voice Security Verification System Based on Millimeter-Wave Radar

M$^{3}$V: A multi-modal multi-view approach for Device-Directed Speech Detection

Millimeter wave gesture recognition using multi-feature fusion models in complex scenes

Speech Acquisition and Recovery Based on Piezoelectric Effect in the Mmwave Band

mmGesture: Semi-supervised gesture recognition system using mmWave radar

mmWave-Whisper: Phone Call Eavesdropping and Transcription Using Millimeter-Wave Radar

A Low-Complexity Hand Gesture Recognition Framework via Dual mmWave FMCW Radar System

WavoID: Robust and Secure Multi-modal User Identification Via Mmwave-Voice Mechanism

Multi-person Device-Free Gesture Recognition Using Mmwave Signals

M-Gesture : Person-Independent Real-Time In-Air Gesture Recognition Using Commodity Millimeter Wave Radar