Abstract:Although speech emotion recognition is challenging, it has broad application prospects in human-computer interaction. Building a system that can accurately and stably recognize emotions from human languages can provide a better user experience. However, the current unimodal emotion feature representations are not distinctive enough to accomplish the recognition, and they do not effectively simulate the inter-modality dynamics in speech emotion recognition tasks. This paper proposes a multimodal method that utilizes both audio and semantic content for speech emotion recognition. The proposed method consists of three parts: two high-level feature extractors for text and audio modalities, and an autoencoder-based feature fusion. For audio modality, we propose a structure called Temporal Global Feature Extractor (TGFE) to extract the high-level features of the time-frequency domain relationship from the original speech signal. Considering that text lacks frequency information, we use only a Bidirectional Long Short-Term Memory network (BLSTM) and attention mechanism to simulate an intra-modal dynamic. Once these steps have been accomplished, the high-level text and audio features are sent to the autoencoder in parallel to learn their shared representation for final emotion classification. We conducted extensive experiments on three public benchmark datasets to evaluate our method. The results on Interactive Emotional Motion Capture (IEMOCAP) and Multimodal EmotionLines Dataset (MELD) outperform the existing method. Additionally, the results of CMU Multi-modal Opinion-level Sentiment Intensity (CMU-MOSI) are competitive. Furthermore, experimental results show that compared to unimodal information and autoencoder-based feature level fusion, the joint multimodal information (audio and text) improves the overall performance and can achieve greater accuracy than simple feature concatenation.

Multi-feature Fusion Speech Emotion Recognition Based on SVM

Emotion Recognition in Videos via Fusing Multimodal Features.

Deep Learning and SVM-based Emotion Recognition from Chinese Speech for Smart Affective Services

Speech emotion recognition based on multi-output GMM and SVM

Speech Emotion Recognition Based on Multi-task Deep Feature Extraction and MKPCA Feature Fusion

Speaker Independent Emotion Recognition Based on SVM/HMMS Fusion System

Visual-Audio Emotion Recognition Based on Multi-Task and Ensemble Learning with Multiple Features

Global-Local-Feature-Fused Driver Speech Emotion Detection for Intelligent Cockpit in Automated Driving.

Multi-modal Emotion Recognition Based on Speech and Image.

Research on Emotion Recognition Method of Flight Training Based on Multimodal Fusion

Multi-Level Speech Emotion Recognition Based on Fisher Criterion and SVM

An autoencoder-based feature level fusion for speech emotion recognition

Speech emotion recognition using multimodal feature fusion with machine learning approach

Multi-Modal Fusion Emotion Recognition Method of Speech Expression Based on Deep Learning

Speech emotion recognition using combination of features

Multi-Modal Emotion Recognition Fusing Video and Audio

Multimodal Emotion Recognition Based on Feature Fusion.

Speech Emotion Recognition based on Optimized Support Vector Machine.

Exploration of Complementary Features for Speech Emotion Recognition Based on Kernel Extreme Learning Machine

Speech Emotion Recognition Based on SVM and GMM-HMM Hybrid System

User Emotion Recognition Method Based on Facial Expression and Speech Signal Fusion