Abstract:Whisper, as a form of speech, is not sufficiently addressed by mainstream speech applications. This is due to the fact that systems built for normal speech do not work as expected for whispered speech. A first step to building a speech application that is inclusive of whispered speech, is the successful classification of whispered speech and normal speech. Such a front-end classification system is expected to have high accuracy and low computational overhead, which is the scope of this paper. One of the characteristics of whispered speech is the absence of the fundamental frequency (or pitch), and hence the pitch harmonics as well. The presence of the pitch and pitch harmonics in normal speech, and its absence in whispered speech, is evident in the spectral envelope of the Fourier transform. We observe that this characteristic is predominant in the first quarter of the spectrum, and exploit the same as a feature. We propose the use of one dimensional convolutional neural networks (1D-CNN) to capture these features from the quartered spectral envelope (QSE). The system yields an accuracy of 99.31% when trained and tested on the wTIMIT dataset, and 100% on the CHAINS dataset. The proposed feature is compared with Mel frequency cepstral coefficients (MFCC), a staple in the speech domain. The proposed classification system is also compared with the state-of-the-art system based on log-filterbank energy (LFBE) features trained on long short-term memory (LSTM) network. The proposed system based on 1D-CNN performs better than, or as good as, the state-of-the-art across multiple experiments. It also converges sooner, with lesser computational overhead. Finally, the proposed system is evaluated under the presence of white noise at various signal-to-noise ratios and found to be robust.

Whisper-to-speech Conversion Using Restricted Boltzmann Machine Arrays

Silent Speech Decoding Using Spectrogram Features Based on Neuromuscular Activities

Modeling spectral envelopes using restricted Boltzmann machines for statistical parametric speech synthesis

ViSPer: A Multilingual TTS Approach Based on VITS Using Deep Feature Loss

Joint Spectral Distribution Modeling Using Restricted Boltzmann Machines For Voice Conversion

Modeling Spectral Envelopes Using Restricted Boltzmann Machines and Deep Belief Networks for Statistical Parametric Speech Synthesis

Voice Conversion Using Conditional Restricted Boltzmann Machine

End-to-End Whisper to Natural Speech Conversion using Modified Transformer Network

Modeling Spectral Envelopes Using Deep Conditional Restricted Boltzmann Machines for Statistical Parametric Speech Synthesis.

Lip Movement Generation Using Restricted Boltzmann Machines For Visual Speech Synthesis

Restricted Boltzmann Machine Based Spectrum Modeling and Unit Selection Speech Synthesis Method

Quartered Spectral Envelope and 1D-CNN-based Classification of Normally Phonated and Whispered Speech

Improving Whispered Speech Recognition Performance using Pseudo-whispered based Data Augmentation

Voice Conversion Using Deep Neural Networks with Layer-Wise Generative Training

A noise-robust voice conversion method with controllable background sounds

BrainECHO: Semantic Brain Signal Decoding through Vector-Quantized Spectrogram Reconstruction for Whisper-Enhanced Text Generation

DQ-Whisper: Joint Distillation and Quantization for Efficient Multilingual Speech Recognition

M2R-Whisper: Multi-stage and Multi-scale Retrieval Augmentation for Enhancing Whisper

Efficient Compression of Multitask Multilingual Speech Models

Whispered-to-voiced Alaryngeal Speech Conversion with Generative Adversarial Networks

Multilingual DistilWhisper: Efficient Distillation of Multi-task Speech Models via Language-Specific Experts