Abstract:Silent speech interfaces (SSIs) have emerged as innovative non-acoustic communication methods, and our previous study demonstrated the significant potential of three-axis accelerometer-based SSIs to identify silently spoken words with high classification accuracy. The developed accelerometer-based SSI with only four accelerometers and a small training dataset outperformed a conventional surface electromyography (sEMG)-based SSI. In this study, motivated by the promising initial results, we investigated the feasibility of synthesizing spoken speech from three-axis accelerometer signals. This exploration aimed to assess the potential of accelerometer-based SSIs for practical silent communication applications. Nineteen healthy individuals participated in our experiments. Five accelerometers were attached to the face to acquire speech-related facial movements while the participants read 270 Korean sentences aloud. For the speech synthesis, we used a convolution-augmented Transformer (Conformer)-based deep neural network model to convert the accelerometer signals into a Mel spectrogram, from which an audio waveform was synthesized using HiFi-GAN. To evaluate the quality of the generated Mel spectrograms, ten-fold cross-validation was performed, and the Mel cepstral distortion (MCD) was chosen as the evaluation metric. As a result, an average MCD of 5.03 ± 0.65 was achieved using four optimized accelerometers based on our previous study. Furthermore, the quality of generated Mel spectrograms was significantly enhanced by adding one more accelerometer attached under the chin, achieving an average MCD of 4.86 ± 0.65 (p < 0.001, Wilcoxon signed-rank test). Although an objective comparison is difficult, these results surpass those obtained using conventional SSIs based on sEMG, electromagnetic articulography, and electropalatography with the fewest sensors and a similar or smaller number of sentences to train the model. Our proposed approach will contribute to the widespread adoption of accelerometer-based SSIs, leveraging the advantages of accelerometers like low power consumption, invulnerability to physiological artifacts, and high portability.

Inductive conformal prediction for silent speech recognition

Quality-aware Aggregated Conformal Prediction for Silent Speech Recognition

Speech neuromuscular decoding based on spectrogram images using conformal predictors with Bi-LSTM.

Silent Speech Recognition based on sEMG and EEG Signals

Hybrid Silent Speech Interface Through Fusion of Electroencephalography and Electromyography

Silent Speech Decoding Using Spectrogram Features Based on Neuromuscular Activities

A novel silent speech recognition approach based on parallel inception convolutional neural network and Mel frequency spectral coefficient

Real-time Speech Emotion Recognition Based on Syllable-Level Feature Extraction

Decoding Silent Speech Commands from Articulatory Movements Through Soft Magnetic Skin and Machine Learning

Reliability-based cleaning of noisy training labels with inductive conformal prediction in multi-modal biomedical data mining

Silent Speech Recognition Based on Surface Electromyography

Silent Speech Recognition Based on High-Density Surface Electromyogram Using Hybrid Neural Networks

An Interpretable and Generalizable Speech Detector Based on a CNN-LSTM Framework

Feature selection of mime speech recognition using surface electromyography data

Silent Speech Recognition Based on Surface Electromyography Using a Few Electrode Sites under the Guidance from High-Density Electrode Arrays.

Decoding Silent Speech Based on High-Density Surface Electromyogram Using Spatiotemporal Neural Network

Silent Speech Eyewear Interface: Silent Speech Recognition Method Using Eyewear and an Ear-Mounted Microphone with Infrared Distance Sensors

Predictive Inference with Feature Conformal Prediction

All-weather, natural silent speech recognition via machine-learning-assisted tattoo-like electronics

Speech synthesis from three-axis accelerometer signals using conformer-based deep neural network

Objective speech intelligibility prediction using a deep learning model with continuous speech-evoked cortical auditory responses