Speech Emotion Recognition Based on Syllable-Level Feature Extraction

Abdul Rehman Javed,Zhentao Liu,Min Wu,Weihua Cao,Cheng-Shan Jiang

DOI: https://doi.org/10.2139/ssrn.4372645

2023-01-01

Abstract:Download This Paper Open PDF in Browser Add Paper to My Library Share: Permalink Using these links will ensure access to this page indefinitely Copy URL Copy DOI

What problem does this paper attempt to address?

Speech Emotion Recognition Based on Syllable-Level Feature Extraction

Abdul Rehman,Zhen-Tao Liu,Min Wu,Wei-Hua Cao,Cheng-Shan Jiang

DOI: https://doi.org/10.1016/j.apacoust.2023.109444

IF: 3.614

2023-01-01

Applied Acoustics

Abstract:Speech emotion recognition systems have high computational requirements for deep learning models and low generalizability mainly because of the poor reliability of emotional measurements across mul-tiple corpora. To solve these problems, we present a speech emotion recognition system based on a reductionist approach of decomposing and analyzing syllable-level features. Mel-spectrogram of an audio stream is decomposed into syllable-level components, which are then analyzed to extract statistical fea-tures. The proposed method uses formant attention, noise-gate filtering, and rolling normalization con-texts to decrease contextual differences and increase focus the attention on the structure of a formant. A set of syllable-level formant features is extracted and fed into a single hidden layer neural network that makes predictions for each syllable as opposed to the conventional approach of using a sophisticated deep learner to make sentence-wide predictions. The syllable level predictions help to lower the aggre-gated error in utterance level cross-corpus predictions. The experiments on IEMOCAP (IE), MSP-Improv (MI), and RAVDESS (RA) databases show that the method archives better than the state-of-the-art cross-corpus unweighted accuracy of 47.6% for IE to MI and 56.2% for MI to IE.& COPY; 2023 Elsevier Ltd. All rights reserved.
Real-time Speech Emotion Recognition Based on Syllable-Level Feature Extraction

Abdul Rehman,Zhentao Liu,Min Wang,Wenming Cao,Cheng-Shan Jiang

DOI: https://doi.org/10.48550/arxiv.2204.11382

2022-01-01

Abstract:Speech emotion recognition systems have high prediction latency because of the high computational requirements for deep learning models and low generalizability mainly because of the poor reliability of emotional measurements across multiple corpora. To solve these problems, we present a speech emotion recognition system based on a reductionist approach of decomposing and analyzing syllable-level features. Mel-spectrogram of an audio stream is decomposed into syllable-level components, which are then analyzed to extract statistical features. The proposed method uses formant attention, noise-gate filtering, and rolling normalization contexts to increase feature processing speed and tolerance to adversity. A set of syllable-level formant features is extracted and fed into a single hidden layer neural network that makes predictions for each syllable as opposed to the conventional approach of using a sophisticated deep learner to make sentence-wide predictions. The syllable level predictions help to achieve the real-time latency and lower the aggregated error in utterance level cross-corpus predictions. The experiments on IEMOCAP (IE), MSP-Improv (MI), and RAVDESS (RA) databases show that the method archives real-time latency while predicting with state-of-the-art cross-corpus unweighted accuracy of 47.6% for IE to MI and 56.2% for MI to IE.
Syllable Level Speech Emotion Recognition Based on Formant Attention

Abdul Rehman,Zhen-Tao Liu,Jin-Meng Xu

DOI: https://doi.org/10.1007/978-3-030-93049-3_22

2021-01-01

Abstract:The performance of speech emotion recognition (SER) systems can be significantly compromised by the sentence structure of words being spoken. Since the relation between affective content and the lexical content of speech is difficult to determine in a small training sample, the temporal sequence based pattern recognition methods fail to generalize over different sentences in the wild. In this paper, a method to recognize emotion for each syllable separately instead of using a pattern recognition for a whole utterance is proposed. The work emphasizes the preprocessing of the received audio samples where the skeleton structure of Mel-spectrum is extracted using formant attention method, then utterances are sliced into syllables based on the contextual changes in the formants. The proposed syllable onset detection and feature extraction method is validated on two databases for the accuracy of emotional class prediction. The suggested SER method achieves up to 67% and 55% unweighted accuracy on IEMOCAP and MSP-Improv datasets, respectively. The effectiveness of the method is proved by the experimentation results and compared to the state-of-the-art SER methods.
Speech Emotion Recognition Based on Formant Characteristics Feature Extraction and Phoneme Type Convergence.

Zhen-Tao Liu,Abdul Rehman,Min Wu,Wei-Hua Cao,Man Hao

DOI: https://doi.org/10.1016/j.ins.2021.02.016

IF: 8.1

2021-01-01

Information Sciences

Abstract:Speech Emotion Recognition (SER) has numerous applications including human-robot interaction, online gaming, and health care assistance. While deep learning-based approaches achieve considerable precision, they often come with high computational and time costs. Indeed, feature learning strategies must search for important features in a large amount of speech data. In order to reduce these time and computational costs, we propose pre-processing step in which speech segments with similar formant characteristics are clustered together and labeled as the same phoneme. The phoneme occurrence rates in emotional utterances are then used as the input features for classifiers. Using six databases (EmoDB, RAVDESS, IEMOCAP, ShEMO, DEMoS and MSP-Improv) for evaluation, the level of accuracy is comparable to that of current state-of-the-art methods and the required training time was significantly reduced from hours to minutes. (c) 2021 Elsevier Inc. All rights reserved.
Pitch envelope based frame level score reweighed algorithm for emotion robust speaker recognition.

Dongdong Li,Yingchun Yang,Ting Huang

DOI: https://doi.org/10.1109/ACII.2009.5349589

2009-01-01

Abstract:Speech with various emotions aggravates the performance of speaker recognition systems. In this paper, a novel score normalization approach called pitch envelope based frame level score reweighted (PFLSR) algorithm is introduced to compensate the influence of the affective speech on speaker recognition. The approach assumes that the maximum likelihood model is not easily changed with the expressive corruption for most of the frames. Thus the test frames are divided into two parts according to F0, the heavily affected ones and the slightly affected ones. The confidences of the slightly affected frames are reweighted into new scores to strengthen their confidence, and to optimize the final accumulated frame scores over the whole test utterance. The experiments are conducted on the Mandarin Affective Speech Corpus. An improvement of 15.1% in identification rate over the traditional speaker recognition is achieved. ©2009 IEEE.
Cost-Sensitive Learning for Emotion Robust Speaker Recognition

Dongdong Li,Yingchun Yang,Weihui Dai

DOI: https://doi.org/10.1155/2014/628516

2014-01-01

Abstract:In the field of information security, voice is one of the most important parts in biometrics. Especially, with the development of voice communication through the Internet or telephone system, huge voice data resources are accessed. In speaker recognition, voiceprint can be applied as the unique password for the user to prove his/her identity. However, speech with various emotions can cause an unacceptably high error rate and aggravate the performance of speaker recognition system. This paper deals with this problem by introducing a cost-sensitive learning technology to reweight the probability of test affective utterances in the pitch envelop level, which can enhance the robustness in emotion-dependent speaker recognition effectively. Based on that technology, a new architecture of recognition system as well as its components is proposed in this paper. The experiment conducted on the Mandarin Affective Speech Corpus shows that an improvement of 8% identification rate over the traditional speaker recognition is achieved.
Exploring Spatio-Temporal Representations by Integrating Attention-based Bidirectional-LSTM-RNNs and FCNs for Speech Emotion Recognition

Ziping Zhao,Yu Zheng,Zixing Zhang,Haishuai Wang,Yiqin Zhao,Chao Li

DOI: https://doi.org/10.21437/interspeech.2018-1477

2018-01-01

Abstract:Automatic emotion recognition from speech, which is an important and challenging task in the field of affective computing, heavily relies on the effectiveness of the speech features for classification. Previous approaches to emotion recognition have mostly focused on the extraction of carefully hand-crafted features. How to model spatio-temporal dynamics for speech emotion recognition effectively is still under active investigation. In this paper, we propose a method to tackle the problem of emotional relevant feature extraction from speech by leveraging Attention-based Bidirectional Long Short-Term Memory Recurrent Neural Networks with fully convolutional networks in order to automatically learn the best spatio-temporal representations of speech signals. The learned high-level features are then fed into a deep neural network (DNN) to predict the final emotion. The experimental results on the Chinese Natural Audio-Visual Emotion Database (CHEAVD) and the Interactive Emotional Dyadic Motion Capture (IEMOCAP) corpora show that our method provides more accurate predictions compared with other existing emotion recognition algorithms.
Speech Emotion Recognition Based on Feature Selection and Extreme Learning Machine Decision Tree

Zhen-Tao Liu,Min Wu,Wei-Hua Cao,Jun-Wei Mao,Jian-Ping Xu,Guan-Zheng Tan

DOI: https://doi.org/10.1016/j.neucom.2017.07.050

IF: 6

2018-01-01

Neurocomputing

Abstract:Feature selection is a crucial step in the development of a system for identifying emotions in speech. Recently, the interaction between features generated from the same audio source was rarely considered, which may produce redundant features and increase the computational costs. To solve this problem, feature selection method based on correlation analysis and Fisher is proposed, which can remove the redundant features that have close correlations with each other. To improve the recognition performance of the feature subset after proposal feature selection further, an emotion recognition method based on extreme learning machine (ELM) decision tree is proposed according to the confusion degree among different basic emotions. A framework of speech emotion recognition is proposed and the classification experiments based on proposed classification method by using Chinese speech database from institute of automation of Chinese academy of sciences (CASIA) are performed. And the experimental results show that the proposal achieved 89.6% recognition rate on average. By proposal, it would be fast and efficient to discriminate emotional states of different speakers from speech, and it would make it possible to realize the interaction between speaker-independent and computer/robot in the future.
Cross-Corpus Speech Emotion Recognition Based on Hybrid Neural Networks

Abdul Rehman,Zhen-Tao Liu,Dan-Yun Li,Bao-Han Wu

DOI: https://doi.org/10.23919/ccc50068.2020.9189368

2020-01-01

Abstract:Speech emotion recognition helps enrich next-generation AI with emotional intelligence abilities by grasping the emotion from voice and words. At the current stage, speech emotion recognition (SER) is only used within experimental boundaries. The current challenge facing the SER research is the lack of robustness across cultures, languages and even minor differences such as age-gaps of speakers. To create a more adaptable SER in adversarial circumstances, we propose hybrid neural networks architecture that creates a holistic model by embedding the Mel Frequency Cepstrum Coefficients as one-hot inputs such that differences in coefficients in each emotional category are inflated according to their importance. We performed experiments on three different databases to test the cross-corpus effectiveness of the proposed model.
Speech Emotion Recognition And Intensity Estimation

Ml Song,C Chen,Jj Bu,My You

DOI: https://doi.org/10.1007/978-3-540-24768-5_43

2004-01-01

Abstract:In this paper, a system for speech emotion analysis is presented. On a corpus of over 1700 utterances from an individual, the feature vector stream is extracted for each utterance based on short time log frequency power coefficients (LFCC). Using the feature vector streams, we trained Hidden Markov Models (HMMs) to recognize seven basic categories emotions: neutral, happiness, anger, sadness, surprise, fear. Furthermore, the intensity of the basic emotion is divided into 3 levels. And we trained 18 sub-HMMs to identify the intensity of the recognized emotions. Experiment result shows that the emotion recognition rate and the estimation of intensity performed by our system are of good and convincing quality.
Speech Emotion Recognition Based on Linear Discriminant Analysis and Support Vector Machine Decision Tree

Jun-Wei Mao,Yong He,Zhen-Tao Liu

DOI: https://doi.org/10.23919/chicc.2018.8482931

2018-01-01

Abstract:For speech emotion recognition, emotional feature set with high dimension may produce redundant features and influence the recognition accuracy. To solve this problem and obtain the optimal emotional feature subset of speech, a feature dimension reduction based on linear discriminant analysis is proposed. According to the confusion degree between different basic emotions, an emotion recognition method based on support vector machine decision tree is proposed. Experiment on speaker-dependent speech emotion recognition using Chinese speech database from institute of automation of Chinese academy of sciences is performed and a speech emotion recognition system is presented, where standard feature sets of the INTER-SPEECH and classic classifiers are used in comparative experiments respectively. Experimental results show that the proposal achieves 84.39% recognition accuracy on average. By proposal, it would be fast and efficient to discriminate emotional states of diverse speakers from speech, and it would make it possible to realize the interaction between speaker and computer/robot in the future.
Speech Emotion Recognition Based on Convolutional Neural Network with Attention-Based Bidirectional Long Short-Term Memory Network and Multi-Task Learning

Zhen-Tao Liu,Meng-Ting Han,Bao-Han Wu,Abdul Rehman

DOI: https://doi.org/10.1016/j.apacoust.2022.109178

IF: 3.614

2023-01-01

Applied Acoustics

Abstract:Speech emotion recognition (SER) is a challenging task since the distribution of the features is different among various people. To improve generalization performance and accuracy of SER, we employ balanced augmented sampling on the triple-channel log-Mel spectrograms to improve the imbalance of the sample distribution among emotional categories and provide sufficient inputs for the deep neural network model. Time-domain filter and frequency-domain filter are used to process the triple-channel log-Mel spectrograms respectively in order to increase the diversity of features. After that, a deep neural network composed of convolutional neural network (CNN) and attention-based bidirectional long short-term memory network (ABLSTM) is employed for feature extraction, in which multi-task learning is adopted to improve the performance of the deep neural network. We select seven auxiliary tasks and determine the optimal auxiliary tasks through experimental comparison. Finally, our method is evaluated on IEMOCAP and MSP-IMPROV database, and it achieves 70.27% and 66.27% in terms of WAR and UAR on IEMOCAP database, while the WAR and UAR are 60.90% and 61.83% on MSP-IMPROV database respectively, which demonstrates its better performance than other works.
Speaker-independent Speech Emotion Recognition Based on Random Forest Feature Selection Algorithm

Wei-Hua Cao,Jian-Ping Xu,Zhen-Tao Liu

DOI: https://doi.org/10.23919/chicc.2017.8029112

2017-01-01

Abstract:Feature selection is a crucial step in the development of a system for identifying emotions in speech. How to select high correlation features is an open question. This paper focuses on feature selection method which aims to extract the most effective acoustic features to improve the performance of emotion recognition. Emotional feature selection of speaker-independent speech based on Random Forest is proposed, which can remove the redundant features that have high correlations with each other. Experiment on the speech emotion recognition based on Support Vector Machines (SVM) is performed, where the speaker-independent features selected by the proposal and Spearman correlation analysis are used for emotion recognition, respectively. Experimental results show that the proposal achieves 78.6% recognition rate on average, which is 2.2% higher than using Spearman.
Speaker-Independent Speech Emotion Recognition Based On Cnn-Blstm And Multiple Svms

Zhen-Tao Liu,Peng Xiao,Dan-Yun Li,Man Hao

DOI: https://doi.org/10.1007/978-3-030-27535-8_43

2019-01-01

Abstract:Speaker-independent speech emotion recognition (SER) is a complex task because of the variations among different speakers, such as gender, age and other emotional irrelevant factors, which may lead to a tremendous difference among emotional features' distribution. To alleviate the adverse effect generated by emotional irrelevant factors, we propose a SER model that consists of convolutional neutral networks (CNN), attention-based bidirectional long short-term memory network (BLSTM), and multiple linear support vector machines. The log Mel-spectrogram with its velocity (delta) and acceleration (double delta) coefficients are adopted as the inputs of our model since they can apply sufficient information for feature learning by our model. Several groups of speaker-independent SER experiments are performed on the Interactive Emotional Dyadic Motion Capture Database (IEMOCAP) database to improve the credibility of the results. Experimental results show that our method obtains unweighted average recall of 61.50% and weighted average recall of 62.31% for speaker-independent SER on IEMOCAP database.
Emotion Recognition From Noisy Speech

Mingyu You,Chun Chen,Jiajun Bu,Jia Liu,Jianhua Tao

DOI: https://doi.org/10.1109/ICME.2006.262865

2006-01-01

Abstract:This paper presents an emotion recognition system from clean and noisy speech. Geodesic distance was adopted to preserve the intrinsic geometry of emotional speech. Based on the geodesic distance estimation, an enhanced Lipschitz embedding was developed to embed the 64-dimensional acoustic features into a six-dimensional space. In order to avoid the problems brought by noise reduction, emotion recognition from noisy speech was performed directly. Linear Discriminant Analysis (LDA), Principal Component Analysis (PCA) and feature selection by Sequential Forward Selection (SFS) with Support Vector Machine (SVM) were also included to compress acoustic features before classifying the emotional states of clean and noisy speech. Experimental results demonstrate that compared with other methods, the proposed system makes approximately 10% improvement. The performance of our system is also robust when speech data is corrupted by increasing noise.
Speech Emotion Recognition Using Acoustic Features

JIANG Danning,CAI Lianhong

DOI: https://doi.org/10.3321/j.issn:1000-0054.2006.01.023

2006-01-01

Abstract:A speech emotion recognition algorithm was developed based on the statistical and temporal features of the acoustic parameters for discriminating between emotions.The system first extracted the basic prosody parameters and spectral parameters,then used a PNN(probabilistic neural network) to model the statistic features and a HMM(hidden Markov model) to model the temporal features.The sum and product rules were used to combine the probabilities from each group of features for the final decision.Experiments on the Chinese speech corpus showed how the statistical and temporal features tend to reflect different aspects of emotions.The accuracy rate obtained by feature combination is higher than that by each group alone,reaching a maximum of(92.9%.)
Fuzzy speech emotion recognition considering semantic awareness

Yu Xiong,Ting Cai,Xin Zhong,Song Zhou,Linqin Cai

DOI: https://doi.org/10.3233/jifs-232280

2024-02-28

Journal of Intelligent & Fuzzy Systems

Abstract:Speech emotion recognition is of great significance in the industry such as social robots, health care, and intelligent education. Due to the obscurity of emotional expression in speech, most works on speech emotion recognition (SER) ignore the consistency of speech emotion recognition, leading to fuzzy expression and low accuracy in emotional recognition. In this paper, we propose a semantic aware speech emotion recognition model to alleviate this issue. Specifically, a speech feature extraction module based on CNN and Transformer is designed to extract local and global information from the speech. Moreover, a semantic embedding support module is proposed to use text semantic information as auxiliary information to assist the model in extracting emotional features of speech, and can effectively overcome the problem of low recognition rate caused by emotional ambiguity. In addition, the model uses a key-value pair attention mechanism to fuse the features, which makes the fusion of speech and text features preferable. In experiments on two benchmark corpora IEMOCAP and EMO-DB, the recognition rates of 74.3% and 72.5% were obtained under respectively, which show that the proposed model can significantly improve the accuracy of emotion recognition.
Paralinguistic and spectral feature extraction for speech emotion classification using machine learning techniques

Tong Liu,Xiaochen Yuan

DOI: https://doi.org/10.1186/s13636-023-00290-x

2023-05-16

Abstract:Emotion plays a dominant role in speech. The same utterance with different emotions can lead to a completely different meaning. The ability to perform various of emotion during speaking is also one of the typical characters of human. In this case, technology trends to develop advanced speech emotion classification algorithms in the demand of enhancing the interaction between computer and human beings. This paper proposes a speech emotion classification approach based on the paralinguistic and spectral features extraction. The Mel-frequency cepstral coefficients (MFCC) are extracted as spectral feature, and openSMILE is employed to extract the paralinguistic feature. The machine learning techniques multi-layer perceptron classifier and support vector machines are respectively applied into the extracted features for the classification of the speech emotions. We have conducted experiments on the Berlin database to evaluate the performance of the proposed approach. Experimental results show that the proposed approach achieves satisfied performances. Comparisons are conducted in clean condition and noisy condition respectively, and the results indicate better performance of the proposed scheme.

engineering, electrical & electronic,acoustics
Speech emotion recognition based on multi-dimensional feature extraction and multi-scale feature fusion

Lingli Yu,Fengjun Xu,Yundong Qu,Kaijun Zhou

DOI: https://doi.org/10.1016/j.apacoust.2023.109752

IF: 3.614

2024-01-01

Applied Acoustics

Abstract:In recent years, convolutional neural networks have been widely applied in the research of speech emotion recognition. However, most existing methods treat speech spectrogram features as images to learn the emotion features, which makes it difficult to capture rich emotion information at different scales. In this paper, a speech emotion recognition method based on multi-dimensional feature extraction and multi-scale feature fusion is proposed. Multi-dimensional features are extracted from Mel-frequency cepstral coefficients using parallel convolutional layers, emotion feature representations are learned at different scales using multi-scale residual convolutional layers, and a global feature fusion module is finally used to integrate local multi-scale emotion features to obtain key global emotion features. Experiments are conducted on the IEMOCAP, CREMA-D, and RAVDESS dataset, and the results show that our proposed network achieves a 3.5%-4% improvement in four commonly used evaluation metrics, including unweighted accuracy and weighted accuracy, compared with the state-of-the-art methods.

acoustics
Multimodal Emotion Recognition from Raw Audio with Sinc-convolution

Xiaohui Zhang,Wenjie Fu,Mangui Liang

2024-02-19

Abstract:Speech Emotion Recognition (SER) is still a complex task for computers with average recall rates usually about 70% on the most realistic datasets. Most SER systems use hand-crafted features extracted from audio signal such as energy, zero crossing rate, spectral information, prosodic, mel frequency cepstral coefficient (MFCC), and so on. More recently, using raw waveform for training neural network is becoming an emerging trend. This approach is advantageous as it eliminates the feature extraction pipeline. Learning from time-domain signal has shown good results for tasks such as speech recognition, speaker verification etc. In this paper, we utilize Sinc-convolution layer, which is an efficient architecture for preprocessing raw speech waveform for emotion recognition, to extract acoustic features from raw audio signals followed by a long short-term memory (LSTM). We also incorporate linguistic features and append a dialogical emotion decoding (DED) strategy. Our approach achieves a weighted accuracy of 85.1\% in four class emotion on the Interactive Emotional Dyadic Motion Capture (IEMOCAP) dataset.

Sound,Multimedia,Audio and Speech Processing

Speech Emotion Recognition Based on Syllable-Level Feature Extraction

Speech Emotion Recognition Based on Syllable-Level Feature Extraction

Real-time Speech Emotion Recognition Based on Syllable-Level Feature Extraction

Syllable Level Speech Emotion Recognition Based on Formant Attention

Speech Emotion Recognition Based on Formant Characteristics Feature Extraction and Phoneme Type Convergence.

Pitch envelope based frame level score reweighed algorithm for emotion robust speaker recognition.

Cost-Sensitive Learning for Emotion Robust Speaker Recognition

Exploring Spatio-Temporal Representations by Integrating Attention-based Bidirectional-LSTM-RNNs and FCNs for Speech Emotion Recognition

Speech Emotion Recognition Based on Feature Selection and Extreme Learning Machine Decision Tree

Cross-Corpus Speech Emotion Recognition Based on Hybrid Neural Networks

Speech Emotion Recognition And Intensity Estimation

Speech Emotion Recognition Based on Linear Discriminant Analysis and Support Vector Machine Decision Tree

Speech Emotion Recognition Based on Convolutional Neural Network with Attention-Based Bidirectional Long Short-Term Memory Network and Multi-Task Learning

Speaker-independent Speech Emotion Recognition Based on Random Forest Feature Selection Algorithm

Speaker-Independent Speech Emotion Recognition Based On Cnn-Blstm And Multiple Svms

Emotion Recognition From Noisy Speech

Speech Emotion Recognition Using Acoustic Features

Fuzzy speech emotion recognition considering semantic awareness

Paralinguistic and spectral feature extraction for speech emotion classification using machine learning techniques

Speech emotion recognition based on multi-dimensional feature extraction and multi-scale feature fusion

Multimodal Emotion Recognition from Raw Audio with Sinc-convolution