Abstract:Speech emotion recognition plays an increasingly important role in emotional computing and is still a challenging task due to its complexity. In this study, we developed a framework integrating three distinctive classifiers: a deep neural network (DNN), a convolution neural network (CNN), and a recurrent neural network (RNN). The framework was used for categorical recognition of four discrete emotions (i.e., angry, happy, neutral and sad). Frame-level low-level descriptors (LLDs), segment-level mel-spectrograms (MS), and utterance-level outputs of high-level statistical functions (HSFs) on LLDs were passed to RNN, CNN, and DNN, separately. Three individual models of LLD-RNN, MS-CNN, and HSF-DNN were obtained. In the models of MS-CNN and LLD-RNN, the attention mechanism based weighted-pooling method was utilized to aggregate the CNN and RNN outputs. To effectively utilize the interdependencies between the two approaches of emotion description (discrete emotion categories and continuous emotion attributes), a multi-task learning strategy was implemented in these three models to acquire generalized features by simultaneously operating classification of discrete categories and regression of continuous attributes. Finally, a confidence-based fusion strategy was developed to integrate the power of different classifiers in recognizing different emotional states. Three experiments on emotion recognition based on the IEMOCAP corpus were conducted. Our experimental results show that the weighted pooling method based on attention mechanism endowed the neural networks with the capability to focus on emotionally salient parts. The generalized features learned in the multi-task learning helped the neural networks to achieve higher accuracies in the tasks of emotion classification. Furthermore, our proposed fusion system achieved weighted accuracy of 57.1% and unweighted accuracy of 58.3%, which were significantly higher than those of each individual classifier. The effectiveness of the proposed approach based on classifier fusion was thus validated.

Speech Emotion Recognition with Emotion-Pair Based Framework Considering Emotion Distribution Information in Dimensional Emotion Space.

Exploring Spatio-Temporal Representations by Integrating Attention-based Bidirectional-LSTM-RNNs and FCNs for Speech Emotion Recognition

MFDR: Multiple-stage Fusion and Dynamically Refined Network for Multimodal Emotion Recognition

Speech Emotion Recognition Based on Linear Discriminant Analysis and Support Vector Machine Decision Tree

Speech Emotion Recognition Based on Feature Selection and Extreme Learning Machine Decision Tree

Hierarchical Framework For Speech Emotion Recognition

Study on Feature Subspace of Archetypal Emotions for Speech Emotion Recognition

Speech Emotion Recognition Based on a Fusion of All-Class and Pairwise-Class Feature Selection

Speech Emotion Recognition Using Fusion of Three Multi-Task Learning-Based Classifiers: HSF-DNN, MS-CNN and LLD-RNN

Ann Based Decision Fusion for Speech Emotion Recognition

An autoencoder-based feature level fusion for speech emotion recognition

Emotion Recognition From Noisy Speech

Speech emotion recognition via discriminant-cascading dimensionality reduction

A Discriminative Feature Representation Method Based on Cascaded Attention Network With Adversarial Strategy for Speech Emotion Recognition

Speech Emotion Recognition Based on Multi-task Deep Feature Extraction and MKPCA Feature Fusion

Manifolds Based Emotion Recognition in Speech.

Combining feature-level and decision-level fusion in a hierarchical classifier for emotion recognition in the wild

Speech Emotion Recognition with Heterogeneous Feature Unification of Deep Neural Network

A Dimensional Emotion Model Driven Multi-stage Classification of Emotional Speech

Multi-stage Classification of Emotional Speech Motivated by a Dimensional Emotion Model

Speech Emotion Recognition using Semantic Information