Abstract:The relationships between muscle movements and neural signals make it possible to decode silent speech based on neuromuscular activities. The decoding can be formulated as a supervised classification task. The electromyography (EMG) captured from surface articulatory muscles contains useful information that can help assist in decoding of speech. Spectrograms obtained from EMG have a wealth of information relating to the decoding, but have not yet been fully explored. In addition, the decoding results are often uncertain. Therefore, it is important to quantify the prediction confidence. This paper aims to improve the decoding performance by representing time series signals as spectrograms and utilising Inductive Conformal Prediction (ICP) to provide predictions with confidence. All EMG data are recorded on six dedicated facial muscles while participants recite the displayed words subvocally. Three pre-trained convolutional models of MobileNet-V1, ResNet18 and Xception are used to extract features from spectrograms for classification. Both bidirectional Long-Short Time Memory (Bi-LSTM) and Gate Recurrent Unit (GRU) classifiers are used for prediction. Furthermore, an ICP decoder based on Bi-LSTM is built to provide guaranteed predictions for each example at a specified confidence level. The proposed method of combining feature extraction based on Xception and classification using Bi-LSTM gives a higher accuracy of 0.87 than other methods. ICP outputs confidence measurements for each example that can help users to evaluate the reliability of new predictions. Experimental results demonstrate the practical usefulness in decoding articulatory neuromuscular activity and the advantages in applying ICP.

Silent Speech Recognition based on sEMG and EEG Signals

Hybrid Silent Speech Interface Through Fusion of Electroencephalography and Electromyography

Silent Speech Decoding Using Spectrogram Features Based on Neuromuscular Activities

Silent Speech Recognition Based on Surface Electromyography

Feature selection of mime speech recognition using surface electromyography data

Inductive conformal prediction for silent speech recognition

Convolutional Neural Network applied in mime speech recognition using sEMG data

Design and implementation of a silent speech recognition system based on sEMG signals: A neural network approach

Attention Bidirectional LSTM Networks Based Mime Speech Recognition Using Semg Data

Speech neuromuscular decoding based on spectrogram images using conformal predictors with Bi-LSTM.

A novel silent speech recognition approach based on parallel inception convolutional neural network and Mel frequency spectral coefficient

Exploration on Channel-interactive Features in Silent Speech Recognition

Quality-aware Aggregated Conformal Prediction for Silent Speech Recognition

sEMG-based technology for silent voice recognition

Preprocessing Improvement in Mime Speech Recognition based on Surface Electromyogram

Silent Speech Recognition Based on High-Density Surface Electromyogram Using Hybrid Neural Networks

Multimodal Speech Recognition Using EEG and Audio Signals: A Novel Approach for Enhancing ASR Systems

Encoder-Decoder Architectures for Silent Speech Recognition Based on High-density Surface Electromyogram

Decoding Silent Speech Based on High-Density Surface Electromyogram Using Spatiotemporal Neural Network

Silent Speech Recognition Based on Surface Electromyography Using a Few Electrode Sites under the Guidance from High-Density Electrode Arrays.

Speech Emotion Recognition Based on Feature Selection and Extreme Learning Machine Decision Tree