Abstract:Speech emotion recognition (SER) has become a crucial topic in the field of human–computer interactions. Feature representation plays an important role in SER, but there are still many challenges in feature representation such as the inability to predict which features are most effective for SER and the cultural differences in emotion expression. Most previous studies use a single type of feature for the recognition task or conduct early fusion of features. However, a single type of feature cannot well reflect the emotions of speech signals. Also, different features contain different information, direct fusion cannot integrate the advantages of different features. To overcome these challenges, this paper proposes a parallel network for multi-scale SER based on a connection attention mechanism (AMSNet). AMSNet fuses fine-grained frame-level manual features with coarse-grained utterance-level deep features. Meanwhile, it adopts different speech emotion feature extraction modules according to the temporal and spatial features of speech signals, which enriches features and improves feature characterization. The network consists of a frame-level representation learning module (FRLM) based on the time structure and an utterance-level representation learning module (URLM) based on the global structure. Besides, improved attention-based long short-term memory (LSTM) is introduced into FRLM to focus on the frames that contribute more to the final emotion recognition result. In URLM, a convolutional neural network with the squeeze-and-excitation block (SCNN) is introduced to extract deep features. In addition, the connection attention mechanism is proposed for feature fusion, which applies different weights to different features. Extensive experiments are conducted on the IEMOCAP and EmoDB datasets, and the results demonstrate the effectiveness and performance superiority of AMSNet. Our code will be publicly available at https://codeocean.com/capsule/8636967/tree/v1.

Spatiotemporal and Frequential Cascaded Attention Networks for Speech Emotion Recognition.

Exploring Spatio-Temporal Representations by Integrating Attention-based Bidirectional-LSTM-RNNs and FCNs for Speech Emotion Recognition

Attention-Enhanced Connectionist Temporal Classification for Discrete Speech Emotion Recognition

Deep Spectrum Feature Representations for Speech Emotion Recognition

Self-attention Transfer Networks for Speech Emotion Recognition

A Discriminative Feature Representation Method Based on Cascaded Attention Network With Adversarial Strategy for Speech Emotion Recognition

Attention Based Fully Convolutional Network for Speech Emotion Recognition

Spontaneous Speech Emotion Recognition Using Multiscale Deep Convolutional LSTM

Speech Emotion Recognition with Multiscale Area Attention and Data Augmentation

Real-time Speech Emotion Recognition Based on Syllable-Level Feature Extraction

Learning Deep Multimodal Affective Features for Spontaneous Speech Emotion Recognition.

A New Network Structure for Speech Emotion Recognition Research

HASTF: a hybrid attention spatio-temporal feature fusion network for EEG emotion recognition

Cascaded Convolutional Recurrent Neural Networks for EEG Emotion Recognition Based on Temporal–Frequency–Spatial Features

Time-Frequency Deep Representation Learning for Speech Emotion Recognition Integrating Self-attention.

Learning multi-scale features for speech emotion recognition with connection attention mechanism

Speech Emotion Recognition Using Multihead Attention in Both Time and Feature Dimensions

Speech Emotion Recognition with Hybrid Neural Network

Representation Learning with Spectro-Temporal-Channel Attention for Speech Emotion Recognition

Temporal Attention Convolutional Network for Speech Emotion Recognition with Latent Representation.

Speech Emotion Recognition Using Attention Model