Abstract:Although speech emotion recognition is challenging, it has broad application prospects in human-computer interaction. Building a system that can accurately and stably recognize emotions from human languages can provide a better user experience. However, the current unimodal emotion feature representations are not distinctive enough to accomplish the recognition, and they do not effectively simulate the inter-modality dynamics in speech emotion recognition tasks. This paper proposes a multimodal method that utilizes both audio and semantic content for speech emotion recognition. The proposed method consists of three parts: two high-level feature extractors for text and audio modalities, and an autoencoder-based feature fusion. For audio modality, we propose a structure called Temporal Global Feature Extractor (TGFE) to extract the high-level features of the time-frequency domain relationship from the original speech signal. Considering that text lacks frequency information, we use only a Bidirectional Long Short-Term Memory network (BLSTM) and attention mechanism to simulate an intra-modal dynamic. Once these steps have been accomplished, the high-level text and audio features are sent to the autoencoder in parallel to learn their shared representation for final emotion classification. We conducted extensive experiments on three public benchmark datasets to evaluate our method. The results on Interactive Emotional Motion Capture (IEMOCAP) and Multimodal EmotionLines Dataset (MELD) outperform the existing method. Additionally, the results of CMU Multi-modal Opinion-level Sentiment Intensity (CMU-MOSI) are competitive. Furthermore, experimental results show that compared to unimodal information and autoencoder-based feature level fusion, the joint multimodal information (audio and text) improves the overall performance and can achieve greater accuracy than simple feature concatenation.

A Novel Dual-Modal Emotion Recognition Algorithm with Fusing Hybrid Features of Audio Signal and Speech Context

Emotion Recognition in Videos via Fusing Multimodal Features.

MFDR: Multiple-stage Fusion and Dynamically Refined Network for Multimodal Emotion Recognition

A Cross-Attention Emotion Recognition Algorithm Based on Audio and Video Modalities

A Novel Emotion-Aware Method Based on the Fusion of Textual Description of Speech, Body Movements, and Facial Expressions

An autoencoder-based feature level fusion for speech emotion recognition

Multi-Modal Emotion Recognition Fusing Video and Audio

Visual-Audio Emotion Recognition Based on Multi-Task and Ensemble Learning with Multiple Features

Multimodal Emotion Recognition Based on Feature Fusion.

A Dual Attention-based Modality-Collaborative Fusion Network for Emotion Recognition

Multimodal Emotion Recognition Based on Multilevel Acoustic and Textual Information

Multimodal emotion recognition from facial expression and speech based on feature fusion

MFGCN: Multimodal fusion graph convolutional network for speech emotion recognition

Multi-modal fusion network with complementarity and importance for emotion recognition

A novel feature fusion network for multimodal emotion recognition from EEG and eye movement signals

Multi-modal fusion emotion recognition based on HMM and ANN

Constructing multi-modal emotion recognition model based on convolutional neural network

Multi-modal Emotion Recognition Based on Speech and Image.

Multi-Modal Fusion Emotion Recognition Method of Speech Expression Based on Deep Learning

Multimodal Emotion Detection via Attention-Based Fusion of Extracted Facial and Speech Features

Enhancing Speech Emotion Recognition Using Dual Feature Extraction Encoders