Abstract:Artificial Neural Network (ANN) models, specifically Convolutional Neural Networks (CNN), were applied to extract emotions based on spectrograms and mel-spectrograms. This study uses spectrograms and mel-spectrograms to investigate which feature extraction method better represents emotions and how big the differences in efficiency are in this context. The conducted studies demonstrated that mel-spectrograms are a better-suited data type for training CNN-based speech emotion recognition (SER). The research experiments employed five popular datasets: Crowd-sourced Emotional Multimodal Actors Dataset (CREMA-D), Ryerson Audio-Visual Database of Emotional Speech and Song (RAVDESS), Surrey Audio-Visual Expressed Emotion (SAVEE), Toronto Emotional Speech Set (TESS), and The Interactive Emotional Dyadic Motion Capture (IEMOCAP). Six different classes of emotions were used: happiness, anger, sadness, fear, disgust, and neutral. However, some experiments were prepared to recognize just four emotions due to the characteristics of the IEMOCAP dataset. A comparison of classification efficiency on different datasets and an attempt to develop a universal model trained using all datasets were also performed. This approach brought an accuracy of 55.89% when recognizing four emotions. The most accurate model for six emotion recognition was trained and achieved 57.42% accuracy on a combination of four datasets (CREMA-D, RAVDESS, SAVEE, TESS). What is more, another study was developed that demonstrated that improper data division for training and test sets significantly influences the test accuracy of CNNs. Therefore, the problem of inappropriate data division between the training and test sets, which affected the results of studies known from the literature, was addressed extensively. The performed experiments employed the popular ResNet18 architecture to demonstrate the reliability of the research results and to show that these problems are not unique to the custom CNN architecture proposed in experiments. Subsequently, the label correctness of the CREMA-D dataset was studied through the employment of a prepared questionnaire.

A Comparative Analysis of Different Approach for Basic Emotions Recognition from Speech

Human-Computer Interaction for Recognizing Speech Emotions Using Multilayer Perceptron Classifier

Human-Computer Interaction with Detection of Speaker Emotions Using Convolution Neural Networks

A Comparison of Machine Learning Algorithms and Feature Sets for Automatic Vocal Emotion Recognition in Speech

Machine learning technique-based emotion classification using speech signals

A Computerized Approach for Automatic Human Emotion Recognition Using Sliding Mode Singular Spectrum Analysis

Optimal Facial Feature Based Emotional Recognition Using Deep Learning Algorithm

Machine learning for human emotion recognition: a comprehensive review

A Comparative Analysis of Machine and Deep Learning Techniques for EEG Evoked Emotion Classification

An efficient algorithm for recognition of emotions from speaker and language independent speech using deep learning

Deep-Net: A Lightweight CNN-Based Speech Emotion Recognition System Using Deep Frequency Features

Emotional Expression Detection in Spoken Language Employing Machine Learning Algorithms

Recognition of Emotions in Speech Using Convolutional Neural Networks on Different Datasets

A Combined CNN Architecture for Speech Emotion Recognition

Comparison of Classical Machine Learning Approaches on Bangla Textual Emotion Analysis

Emotion Recognition from Speech based on Relevant Feature and Majority Voting

Speech emotion analysis using convolutional neural network (CNN) and gamma classifier-based error correcting output codes (ECOC)

Speech Emotion Recognition Using Attention Model

A Methodical Framework Utilizing Transforms and Biomimetic Intelligence-Based Optimization with Machine Learning for Speech Emotion Recognition

Emotion Classification: How Does an Automated System Compare to Naive Human Coders?

Human–Computer Interaction with a Real-Time Speech Emotion Recognition with Ensembling Techniques 1D Convolution Neural Network and Attention