Abstract:Speech emotion recognition (SER) has become a crucial topic in the field of human–computer interactions. Feature representation plays an important role in SER, but there are still many challenges in feature representation such as the inability to predict which features are most effective for SER and the cultural differences in emotion expression. Most previous studies use a single type of feature for the recognition task or conduct early fusion of features. However, a single type of feature cannot well reflect the emotions of speech signals. Also, different features contain different information, direct fusion cannot integrate the advantages of different features. To overcome these challenges, this paper proposes a parallel network for multi-scale SER based on a connection attention mechanism (AMSNet). AMSNet fuses fine-grained frame-level manual features with coarse-grained utterance-level deep features. Meanwhile, it adopts different speech emotion feature extraction modules according to the temporal and spatial features of speech signals, which enriches features and improves feature characterization. The network consists of a frame-level representation learning module (FRLM) based on the time structure and an utterance-level representation learning module (URLM) based on the global structure. Besides, improved attention-based long short-term memory (LSTM) is introduced into FRLM to focus on the frames that contribute more to the final emotion recognition result. In URLM, a convolutional neural network with the squeeze-and-excitation block (SCNN) is introduced to extract deep features. In addition, the connection attention mechanism is proposed for feature fusion, which applies different weights to different features. Extensive experiments are conducted on the IEMOCAP and EmoDB datasets, and the results demonstrate the effectiveness and performance superiority of AMSNet. Our code will be publicly available at https://codeocean.com/capsule/8636967/tree/v1.

Multi-level attention fusion network assisted by relative entropy alignment for multimodal speech emotion recognition

MFDR: Multiple-stage Fusion and Dynamically Refined Network for Multimodal Emotion Recognition

Speech Emotion Recognition Using Multi-Modal Feature Fusion Network

Visual-Audio Emotion Recognition Based on Multi-Task and Ensemble Learning with Multiple Features

Multimodal Speech Emotion Recognition Based on Multi-Scale MFCCs and Multi-View Attention Mechanism

An autoencoder-based feature level fusion for speech emotion recognition

Multi-head attention fusion networks for multi-modal speech emotion recognition

Speech Emotion Recognition by Combining a Unified First-Order Attention Network with Data Balance

WavFusion: Towards wav2vec 2.0 Multimodal Speech Emotion Recognition

Speech Emotion Recognition Via Multi-Level Attention Network

MFGCN: Multimodal fusion graph convolutional network for speech emotion recognition

Emotion Recognition with Multimodal Transformer Fusion Framework Based on Acoustic and Lexical Information

Enhancing Modal Fusion by Alignment and Label Matching for Multimodal Emotion Recognition

A Dual Attention-based Modality-Collaborative Fusion Network for Emotion Recognition

Multimodal Emotion Recognition Based on Feature Selection and Extreme Learning Machine in Video Clips.

Learning multi-scale features for speech emotion recognition with connection attention mechanism

Multi-level Fusion of Wav2vec 2.0 and BERT for Multimodal Emotion Recognition

Emotion Recognition Model Based on Multimodal Decision Fusion

Multi-Modal Fusion Emotion Recognition Method of Speech Expression Based on Deep Learning

Multi-type Features Separating Fusion Learning for Speech Emotion Recognition.

Multimodal emotion recognition based on audio and text by using hybrid attention networks