Abstract:It is encouraged to see that the deep neural networks based speech emotion recognition (DNN-SER) models have achieved the state-of-the-art on public datasets. However, the performance of DNN-SER models is limited due to the following reasons: insufficient training data, emotion ambiguity and class imbalance. Studies show that, without large-scale training data, it is hard for DNN-SER model with cross-entropy loss to learn discriminative features by mapping the speech segments to their category labels. In this study, we propose a deep metric learning based DNN-SER model to facilitate the discriminative feature learning by constraining the feature embeddings in the feature space. For the proof of the concept, we take a four-hidden layer DNN as our backbone for implementation simplicity. Specifically, an emotion identity matrix is formed using one-hot label vectors as supervision information while the emotion embedding matrix is formed using the embedding vectors generated by DNN. An affinity loss is designed based on the above two matrices to simultaneously maximize the inter-class separability and intra-class compactness of the embeddings. Moreover, to restrain the class imbalance problem, the focal loss is introduced to reduce the adverse effect of the majority well-classified samples and gain more focus on the minority misclassified ones. Our proposed DNNSER model is jointly trained using affinity loss and focal loss. Extensive experiments have been conducted on two well-known emotional speech datasets, EMO-DB and IEMOCAP. Compared to DNN-SER baseline, the unweighted accuracy (UA) on EMO-DB and IEMOCAP increased relatively by 10.19% and 10% respectively. Besides, from the confusion matrix of the test results on Emo-DB, it is noted that the accuracy of the most confusing emotion category, 'Happiness', increased relatively by 33.17% and the accuracy of the emotion category with the fewest samples, 'Disgust', increased relatively by 13.62%. These results validate the effectiveness of our proposed DNN-SER model and give the evidence that affinity loss and focal loss help to learn better discriminative features.

Speech Emotion Recognition Based on Discriminative Features Extraction

Attention-Enhanced Connectionist Temporal Classification for Discrete Speech Emotion Recognition

Deep Spectrum Feature Representations for Speech Emotion Recognition

Exploring Spatio-Temporal Representations by Integrating Attention-based Bidirectional-LSTM-RNNs and FCNs for Speech Emotion Recognition

Speech Emotion Recognition Based on Formant Characteristics Feature Extraction and Phoneme Type Convergence.

Speech Emotion Recognition Based on Convolutional Neural Network with Attention-Based Bidirectional Long Short-Term Memory Network and Multi-Task Learning

Speaker-Independent Speech Emotion Recognition Based On Cnn-Blstm And Multiple Svms

Speech Emotion Recognition Using RA-Gmlp Model on Time–Frequency Domain Features Extracted by TFCM

Time-Frequency Attention for Speech Emotion Recognition with Squeeze-and-Excitation Blocks

Speech Emotion Recognition Based on Feature Selection and Extreme Learning Machine Decision Tree

Speech Emotion Recognition Based on Linear Discriminant Analysis and Support Vector Machine Decision Tree

SDTF-Net: Static and Dynamic Time–frequency Network for Speech Emotion Recognition

Speech Emotion Recognition Based on Low-Level Auto-Extracted Time-Frequency Features

Speech Emotion Recognition Based on Syllable-Level Feature Extraction

Temporal-Frequency State Space Duality: An Efficient Paradigm for Speech Emotion Recognition

Speech Emotion Recognition by Combining a Unified First-Order Attention Network with Data Balance

Design of smart home system speech emotion recognition model based on ensemble deep learning and feature fusion

Discriminative Feature Learning For Speech Emotion Recognition

A Discriminative Feature Representation Method Based on Cascaded Attention Network With Adversarial Strategy for Speech Emotion Recognition

Speech Emotion Recognition Via Heterogeneous Feature Learning

Speech Emotion Recognition Using Dual-Stream Representation and Cross-Attention Fusion